Web Scraping with Python: Collecting More Data from the Modern Web

Preface......Page 5
 Why Web Scraping?......Page 6
 About This Book......Page 8
 Conventions Used in This Book......Page 10
 Using Code Examples......Page 11
 How to Contact Us......Page 12
 Acknowledgments......Page 13
I. Building Scrapers......Page 14
 Connecting......Page 15
 An Introduction to BeautifulSoup......Page 18
  Installing BeautifulSoup......Page 19
  Running BeautifulSoup......Page 22
  Connecting Reliably and Handling Exceptions......Page 25
 You Don’t Always Need a Hammer......Page 29
 Another Serving of BeautifulSoup......Page 31
  find() and find_all() with BeautifulSoup......Page 33
  Navigating Trees......Page 36
 Regular Expressions......Page 42
 Regular Expressions and BeautifulSoup......Page 47
 Lambda Expressions......Page 49
 Traversing a Single Domain......Page 52
 Crawling an Entire Site......Page 57
  Collecting Data Across an Entire Site......Page 60
 Crawling Across the Internet......Page 63
 Planning and Defining Objects......Page 70
 Dealing with Different Website Layouts......Page 75
  Crawling Sites Through Search......Page 81
  Crawling Sites Through Links......Page 85
  Crawling Multiple Page Types......Page 88
 Thinking About Web Crawler Models......Page 90
5. Scrapy......Page 92
  Initializing a New Spider......Page 93
 Writing a Simple Scraper......Page 94
 Spidering with Rules......Page 96
 Creating Items......Page 101
 The Item Pipeline......Page 104
 Logging with Scrapy......Page 108
 More Resources......Page 109
 Media Files......Page 111
 Storing Data to CSV......Page 115
 MySQL......Page 117
  Installing MySQL......Page 118
  Some Basic Commands......Page 121
  Integrating with Python......Page 125
  Database Techniques and Good Practice......Page 128
  “Six Degrees” in MySQL......Page 132
 Email......Page 135
II. Advanced Scraping......Page 139
 Document Encoding......Page 140
 Text......Page 141
  Text Encoding and the Global Internet......Page 142
  Reading CSV Files......Page 147
 PDF......Page 150
 Microsoft Word and .docx......Page 152
 Cleaning in Code......Page 158
  Data Normalization......Page 163
  OpenRefine......Page 165
9. Reading and Writing Natural Languages......Page 171
 Summarizing Data......Page 172
 Markov Models......Page 176
  Six Degrees of Wikipedia: Conclusion......Page 181
 Natural Language Toolkit......Page 184
  Installation and Setup......Page 185
  Statistical Analysis with NLTK......Page 186
  Lexicographical Analysis with NLTK......Page 189
 Additional Resources......Page 193
 Python Requests Library......Page 195
 Submitting a Basic Form......Page 196
 Radio Buttons, Checkboxes, and Other Inputs......Page 199
 Submitting Files and Images......Page 200
 Handling Logins and Cookies......Page 201
  HTTP Basic Access Authentication......Page 203
 Other Form Problems......Page 205
 A Brief Introduction to JavaScript......Page 206
  Common JavaScript Libraries......Page 208
 Ajax and Dynamic HTML......Page 210
  Executing JavaScript in Python with Selenium......Page 212
  Additional Selenium Webdrivers......Page 219
 Handling Redirects......Page 220
 A Final Note on JavaScript......Page 222
 A Brief Introduction to APIs......Page 224
  HTTP Methods and APIs......Page 226
  More About API Responses......Page 228
 Parsing JSON......Page 230
 Undocumented APIs......Page 231
  Finding Undocumented APIs......Page 234
  Documenting Undocumented APIs......Page 235
  Finding and Documenting APIs Automatically......Page 236
 Combining APIs with Other Data Sources......Page 239
 More About APIs......Page 244
13. Image Processing and Text Recognition......Page 245
  Pillow......Page 246
  Tesseract......Page 247
  NumPy......Page 250
 Processing Well-Formatted Text......Page 251
  Adjusting Images Automatically......Page 254
  Scraping Text from Images on Websites......Page 258
 Reading CAPTCHAs and Training Tesseract......Page 261
  Training Tesseract......Page 263
 Retrieving CAPTCHAs and Submitting Solutions......Page 268
14. Avoiding Scraping Traps......Page 272
 A Note on Ethics......Page 273
  Adjust Your Headers......Page 274
  Handling Cookies with JavaScript......Page 276
  Timing Is Everything......Page 279
  Hidden Input Field Values......Page 280
  Avoiding Honeypots......Page 282
 The Human Checklist......Page 284
15. Testing Your Website with Scrapers......Page 287
  What Are Unit Tests?......Page 288
 Python unittest......Page 289
  Testing Wikipedia......Page 291
  Interacting with the Site......Page 295
 unittest or Selenium?......Page 300
16. Web Crawling in Parallel......Page 302
 Multithreaded Crawling......Page 303
  Race Conditions and Queues......Page 306
  The threading Module......Page 310
 Multiprocess Crawling......Page 313
  Multiprocess Crawling......Page 315
  Communicating Between Processes......Page 317
 Multiprocess Crawling—Another Approach......Page 320
 Why Use Remote Servers?......Page 322
  Avoiding IP Address Blocking......Page 323
  Portability and Extensibility......Page 324
 Tor......Page 325
  PySocks......Page 326
  Running from a Website-Hosting Account......Page 327
  Running from the Cloud......Page 329
 Additional Resources......Page 331
 Trademarks, Copyrights, Patents, Oh My!......Page 332
  Copyright Law......Page 334
 Trespass to Chattels......Page 335
 The Computer Fraud and Abuse Act......Page 338
 robots.txt and Terms of Service......Page 339
  eBay versus Bidder’s Edge and Trespass to Chattels......Page 344
  United States v. Auernheimer and The Computer Fraud and Abuse Act......Page 346
  Field v. Google: Copyright and robots.txt......Page 349
 Moving Forward......Page 350
Index......Page 352