Web Scraping with Python: Data Extraction from the Modern Web

Cover
Copyright
Table of Contents
Preface
	What Is Web Scraping?
	Why Web Scraping?
	About This Book
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Part I. Building Scrapers
	Chapter 1. How the Internet Works
		Networking
			Physical Layer
			Data Link Layer
			Network Layer
			Transport Layer
			Session Layer
			Presentation Layer
			Application Layer
		HTML
		CSS
		JavaScript
		Watching Websites with Developer Tools
	Chapter 2. The Legalities and Ethics of Web Scraping
		Trademarks, Copyrights, Patents, Oh My!
			Copyright Law
		Trespass to Chattels
		The Computer Fraud and Abuse Act
		robots.txt and Terms of Service
		Three Web Scrapers
			eBay v. Bidder’s Edge and Trespass to Chattels
			United States v. Auernheimer and the Computer Fraud and Abuse Act
			Field v. Google: Copyright and robots.txt
	Chapter 3. Applications of Web Scraping
		Classifying Projects
		E-commerce
			Marketing
		Academic Research
		Product Building
		Travel
		Sales
		SERP Scraping
	Chapter 4. Writing Your First Web Scraper
		Installing and Using Jupyter
		Connecting
		An Introduction to BeautifulSoup
			Installing BeautifulSoup
			Running BeautifulSoup
			Connecting Reliably and Handling Exceptions
	Chapter 5. Advanced HTML Parsing
		Another Serving of BeautifulSoup
			find() and find_all() with BeautifulSoup
			Other BeautifulSoup Objects
			Navigating Trees
		Regular Expressions
		Regular Expressions and BeautifulSoup
		Accessing Attributes
		Lambda Expressions
		You Don’t Always Need a Hammer
	Chapter 6. Writing Web Crawlers
		Traversing a Single Domain
		Crawling an Entire Site
			Collecting Data Across an Entire Site
		Crawling Across the Internet
	Chapter 7. Web Crawling Models
		Planning and Defining Objects
		Dealing with Different Website Layouts
		Structuring Crawlers
			Crawling Sites Through Search
			Crawling Sites Through Links
			Crawling Multiple Page Types
		Thinking About Web Crawler Models
	Chapter 8. Scrapy
		Installing Scrapy
			Initializing a New Spider
		Writing a Simple Scraper
		Spidering with Rules
		Creating Items
		Outputting Items
		The Item Pipeline
		Logging with Scrapy
		More Resources
	Chapter 9. Storing Data
		Media Files
		Storing Data to CSV
		MySQL
			Installing MySQL
			Some Basic Commands
			Integrating with Python
			Database Techniques and Good Practice
			“Six Degrees” in MySQL
		Email
Part II. Advanced Scraping
	Chapter 10. Reading Documents
		Document Encoding
		Text
			Text Encoding and the Global Internet
		CSV
			Reading CSV Files
		PDF
		Microsoft Word and .docx
	Chapter 11. Working with Dirty Data
		Cleaning Text
		Working with Normalized Text
		Cleaning Data with Pandas
			Cleaning
			Indexing, Sorting, and Filtering
			More About Pandas
	Chapter 12. Reading and Writing Natural Languages
		Summarizing Data
		Markov Models
			Six Degrees of Wikipedia: Conclusion
		Natural Language Toolkit
			Installation and Setup
			Statistical Analysis with NLTK
			Lexicographical Analysis with NLTK
		Additional Resources
	Chapter 13. Crawling Through Forms and Logins
		Python Requests Library
		Submitting a Basic Form
		Radio Buttons, Checkboxes, and Other Inputs
		Submitting Files and Images
		Handling Logins and Cookies
			HTTP Basic Access Authentication
		Other Form Problems
	Chapter 14. Scraping JavaScript
		A Brief Introduction to JavaScript
			Common JavaScript Libraries
		Ajax and Dynamic HTML
		Executing JavaScript in Python with Selenium
			Installing and Running Selenium
			Selenium Selectors
			Waiting to Load
			XPath
		Additional Selenium WebDrivers
		Handling Redirects
		A Final Note on JavaScript
	Chapter 15. Crawling Through APIs
		A Brief Introduction to APIs
			HTTP Methods and APIs
			More About API Responses
		Parsing JSON
		Undocumented APIs
			Finding Undocumented APIs
			Documenting Undocumented APIs
		Combining APIs with Other Data Sources
		More About APIs
	Chapter 16. Image Processing and Text Recognition
		Overview of Libraries
			Pillow
			Tesseract
			NumPy
		Processing Well-Formatted Text
			Adjusting Images Automatically
			Scraping Text from Images on Websites
		Reading CAPTCHAs and Training Tesseract
			Training Tesseract
		Retrieving CAPTCHAs and Submitting Solutions
	Chapter 17. Avoiding Scraping Traps
		A Note on Ethics
		Looking Like a Human
			Adjust Your Headers
			Handling Cookies with JavaScript
			TLS Fingerprinting
			Timing Is Everything
		Common Form Security Features
			Hidden Input Field Values
			Avoiding Honeypots
		The Human Checklist
	Chapter 18. Testing Your Website with Scrapers
		An Introduction to Testing
			What Are Unit Tests?
		Python unittest
			Testing Wikipedia
		Testing with Selenium
			Interacting with the Site
	Chapter 19. Web Scraping in Parallel
		Processes Versus Threads
		Multithreaded Crawling
			Race Conditions and Queues
			More Features of the Threading Module
		Multiple Processes
			Multiprocess Crawling
			Communicating Between Processes
		Multiprocess Crawling—Another Approach
	Chapter 20. Web Scraping Proxies
		Why Use Remote Servers?
			Avoiding IP Address Blocking
			Portability and Extensibility
		Tor
			PySocks
		Remote Hosting
			Running from a Website-Hosting Account
			Running from the Cloud
			Moving Forward
		Web Scraping Proxies
			ScrapingBee
			ScraperAPI
			Oxylabs
			Zyte
		Additional Resources
Index
Colophon