Web Scraping with Python: Collecting Data from the Modern Web

Preface
	What Is Web Scraping?
	Why Web Scraping?
	About This Book
	Conventions Used in This Book
	Using Code Examples
	Safari® Books Online
	How to Contact Us
	Acknowledgments
I. Building Scrapers
1. Your First Web Scraper
	Connecting
	An Introduction to BeautifulSoup
		Installing BeautifulSoup
		Running BeautifulSoup
		Connecting Reliably
2. Advanced HTML Parsing
	You Don’t Always Need a Hammer
	Another Serving of BeautifulSoup
		find() and findAll() with BeautifulSoup
		Other BeautifulSoup Objects
		Navigating Trees
			Dealing with children and other descendants
			Dealing with siblings
			Dealing with your parents
	Regular Expressions
	Regular Expressions and BeautifulSoup
	Accessing Attributes
	Lambda Expressions
	Beyond BeautifulSoup
3. Starting to Crawl
	Traversing a Single Domain
	Crawling an Entire Site
		Collecting Data Across an Entire Site
	Crawling Across the Internet
	Crawling with Scrapy
4. Using APIs
	How APIs Work
	Common Conventions
		Methods
		Authentication
	Responses
		API Calls
	Echo Nest
		A Few Examples
	Twitter
		Getting Started
		A Few Examples
	Google APIs
		Getting Started
		A Few Examples
	Parsing JSON
	Bringing It All Back Home
	More About APIs
5. Storing Data
	Media Files
	Storing Data to CSV
	MySQL
		Installing MySQL
		Some Basic Commands
		Integrating with Python
		Database Techniques and Good Practice
		“Six Degrees” in MySQL
	Email
6. Reading Documents
	Document Encoding
	Text
		Text Encoding and the Global Internet
			A brief overview of encoding types
			Encodings in action
	CSV
		Reading CSV Files
	PDF
	Microsoft Word and .docx
II. Advanced Scraping
7. Cleaning Your Dirty Data
	Cleaning in Code
		Data Normalization
	Cleaning After the Fact
		OpenRefine
			Installation
			Using OpenRefine
				Filtering
				Cleaning
8. Reading and Writing Natural Languages
	Summarizing Data
	Markov Models
		Six Degrees of Wikipedia: Conclusion
	Natural Language Toolkit
		Installation and Setup
		Statistical Analysis with NLTK
		Lexicographical Analysis with NLTK
	Additional Resources
9. Crawling Through Forms and Logins
	Python Requests Library
	Submitting a Basic Form
	Radio Buttons, Checkboxes, and Other Inputs
	Submitting Files and Images
	Handling Logins and Cookies
		HTTP Basic Access Authentication
	Other Form Problems
10. Scraping JavaScript
	A Brief Introduction to JavaScript
		Common JavaScript Libraries
			jQuery
			Google Analytics
			Google Maps
	Ajax and Dynamic HTML
		Executing JavaScript in Python with Selenium
	Handling Redirects
11. Image Processing and Text Recognition
	Overview of Libraries
		Pillow
		Tesseract
			Installing Tesseract
		NumPy
	Processing Well-Formatted Text
		Scraping Text from Images on Websites
	Reading CAPTCHAs and Training Tesseract
		Training Tesseract
	Retrieving CAPTCHAs and Submitting Solutions
12. Avoiding Scraping Traps
	A Note on Ethics
	Looking Like a Human
		Adjust Your Headers
		Handling Cookies
		Timing Is Everything
	Common Form Security Features
		Hidden Input Field Values
		Avoiding Honeypots
	The Human Checklist
13. Testing Your Website with Scrapers
	An Introduction to Testing
		What Are Unit Tests?
	Python unittest
		Testing Wikipedia
	Testing with Selenium
		Interacting with the Site
			Drag and drop
			Taking screenshots
	Unittest or Selenium?
14. Scraping Remotely
	Why Use Remote Servers?
		Avoiding IP Address Blocking
		Portability and Extensibility
	Tor
		PySocks
	Remote Hosting
		Running from a Website Hosting Account
		Running from the Cloud
	Additional Resources
	Moving Forward
A. Python at a Glance
	Installation and “Hello, World!”
B. The Internet at a Glance
C. The Legalities and Ethics of Web Scraping
	Trademarks, Copyrights, Patents, Oh My!
		Copyright Law
	Trespass to Chattels
	The Computer Fraud and Abuse Act
	robots.txt and Terms of Service
	Three Web Scrapers
		eBay versus Bidder’s Edge and Trespass to Chattels
		United States v. Auernheimer and The Computer Fraud and Abuse Act
		Field v. Google: Copyright and robots.txt
Index