Webbots, Spiders, and Screen Scrapers

Tables of Contents......Page 8
Introduction......Page 21
The Problem with Browsers......Page 22
Leverage Existing Scripts......Page 23
About the Code......Page 24
Hardware......Page 25
A Disclaimer (This Is Important)......Page 26
PART I: Fundamental Concepts and Techniques......Page 27
Uncovering the Internet’s True Potential......Page 29
Webbot Developers Are in Demand......Page 30
Customize the Internet for Your Business......Page 31
Final Thoughts......Page 32
Inspiration from Browser Limitations......Page 33
Webbots That Aggregate and Filter Information for Relevance......Page 34
Webbots That Act on Your Behalf......Page 35
A Few Crazy Ideas to Get You Started......Page 36
Protect Intellectual Property......Page 37
Create an Online Clipping Service......Page 38
Track Web Technologies......Page 39
Final Thoughts......Page 40
3: Downloading Web Pages......Page 41
Think About Files, Not Web Pages......Page 42
Downloading Files with fopen() and fgets()......Page 43
Downloading Files with file()......Page 45
Basic Authentication......Page 46
Agent Name Spoofing......Page 47
LIB_http......Page 48
Using LIB_http......Page 49
Learning More About HTTP Headers......Page 52
Examining LIB_http’s Source Code......Page 53
Final Thoughts......Page 54
Parsing Poorly Written HTML......Page 55
Using LIB_parse......Page 56
Parsing Text Between Delimiters: return_between()......Page 57
Parsing a Data Set into an Array: parse_array()......Page 59
Parsing Attribute Values: get_attribute()......Page 60
Removing Unwanted Text: remove()......Page 61
Detecting Whether a String Is Within Another String......Page 62
Parsing Unformatted Text......Page 63
Parse in Small Steps......Page 64
Use Regular Expressions Sparingly......Page 65
5: Automating Form Submission......Page 67
Reverse Engineering Form Interfaces......Page 68
Form Handlers......Page 69
Methods......Page 70
Cookies Aren’t Included in the Form, but Can Affect Operation......Page 73
Analyzing a Form......Page 74
Don’t Blow Your Cover......Page 77
Avoid Form Errors......Page 78
Organizing Data......Page 81
Naming Conventions......Page 82
Storing Data in Structured Files......Page 83
Storing Text in a Database......Page 84
Storing Images in a Database......Page 87
Storing References to Image Files......Page 89
Compressing Data......Page 90
Removing Formatting......Page 92
Thumbnailing Images......Page 93
Final Thoughts......Page 94
PART II: Projects......Page 95
7: Price-Monitoring Webbots......Page 97
The Target......Page 98
Initialization and Downloading the Target......Page 99
Further Exploration......Page 103
8: Image-Capturing Webbots......Page 105
Creating the Image-Capturing Webbot......Page 106
Binary-Safe Download Routine......Page 108
The Main Script......Page 109
Final Thoughts......Page 112
Initializing the Webbot and Downloading the Target......Page 113
Setting the Page Base......Page 114
Running a Verification Loop......Page 115
Generating Fully Resolved URLs......Page 116
Displaying the Page Status......Page 117
LIB_http_codes......Page 118
Further Exploration......Page 119
Anonymity with Proxies......Page 121
Non-proxied Environments......Page 122
Your Online Exposure......Page 123
Proxied Environments......Page 124
The Anonymizer Project......Page 125
Writing the Anonymizer......Page 126
Final Thoughts......Page 129
11: Search-Ranking Webbots......Page 131
Description of a Search Result Page......Page 132
How the Search-Ranking Webbot Works......Page 134
Initializing Variables......Page 135
Starting the Loop......Page 136
Parsing the Search Results......Page 137
Spidering Search Engines Is a Bad Idea......Page 140
Further Exploration......Page 141
12: Aggregation Webbots......Page 143
Choosing Data Sources for Webbots......Page 144
Familiarizing Yourself with RSS Feeds......Page 145
Writing the Aggregation Webbot......Page 147
Adding Filtering to Your Aggregation Webbot......Page 149
Further Exploration......Page 151
13: FTP Webbots......Page 153
Example FTP Webbot......Page 154
PHP and FTP......Page 156
Further Exploration......Page 157
14: NNTP News Webbots......Page 159
NNTP Use and History......Page 160
Identifying Newsgroups......Page 161
Finding Articles in Newsgroups......Page 164
Reading an Article from a Newsgroup......Page 165
Further Exploration......Page 167
15: Webbots That Read Email......Page 169
Reading Mail from a POP3 Mail Server......Page 170
Executing POP3 Commands with a Webbot......Page 173
Email-Controlled Webbots......Page 175
Email Interfaces......Page 176
Email, Webbots, and Spam......Page 177
Configuring PHP to Send Mail......Page 178
Sending an Email with mail()......Page 179
Writing a Webbot That Sends Email Notifications......Page 181
Keeping Legitimate Mail out of Spam Filters......Page 182
Sending HTML-Formatted Email......Page 183
Using Returned Emails to Prune Access Lists......Page 184
Writing Webbots That Send Text Messages......Page 185
17: Converting a Website into a Function......Page 187
Writing a Function Interface......Page 188
Analyzing the Target Web Page......Page 189
Using describe_zipcode()......Page 191
Distributing Resources......Page 193
Designing a Custom Lightweight “Web Service”......Page 194
PART III: Advanced Technical Considerations......Page 195
18: Spiders......Page 197
How Spiders Work......Page 198
Example Spider......Page 199
LIB_simple_spider......Page 200
harvest_links()......Page 201
get_domain()......Page 202
exclude_link()......Page 203
Experimenting with the Spider......Page 204
Save Links in a Database......Page 205
Regulate Page Requests......Page 206
19: Procurement Webbots and Snipers......Page 207
Get Purchase Criteria......Page 208
Make Purchase......Page 209
Get Purchase Criteria......Page 210
Synchronize Clocks......Page 211
Further Exploration......Page 213
Final Thoughts......Page 214
20: Webbots and Cryptography......Page 215
Encryption and PHP/CURL......Page 216
Local Certificates......Page 217
Final Thoughts......Page 218
What Is Authentication?......Page 219
Strengthening Authentication by Combining Techniques......Page 220
Basic Authentication......Page 221
Authentication with Cookie Sessions......Page 224
Authentication with Query Sessions......Page 227
Final Thoughts......Page 230
How Cookies Work......Page 231
PHP/CURL and Cookies......Page 233
Purging Temporary Cookies......Page 234
Managing Multiple Users’ Cookies......Page 235
Further Exploration......Page 236
23: Scheduling Webbots and Spiders......Page 237
Preparing Your Webbots to Run as Scheduled Tasks......Page 238
Scheduling a Webbot to Run Daily......Page 239
Complex Schedules......Page 241
Non-Calendar-Based Triggers......Page 242
Final Thoughts......Page 243
Add Variety to Your Schedule......Page 244
PART IV: Larger Considerations......Page 245
Why Design a Stealthy Webbot?......Page 247
Log Files......Page 248
Log-Monitoring Software......Page 250
Run Your Webbot During Busy Hours......Page 251
Final Thoughts......Page 252
25: Writing Fault-Tolerant Webbots......Page 255
Adapting to Changes in URLs......Page 256
Adapting to Changes in Page Content......Page 261
Adapting to Changes in Forms......Page 262
Adapting to Network Outages and Network Congestion......Page 264
Error Handlers......Page 265
Optimizing Web Pages for Search Engine Spiders......Page 267
Title Tags......Page 268
Header Tags......Page 269
JavaScript......Page 270
XML......Page 271
Lightweight Data Exchange......Page 273
SOAP......Page 275
27: Killing Spiders......Page 277
Create a Terms of Service Agreement......Page 278
Use the robots.txt File......Page 279
Selectively Allow Access to Specific Web Agents......Page 280
Use Cookies, Encryption, JavaScript, and Redirection......Page 281
Embed Text in Other Media......Page 282
Fun Things to Do with Unwanted Spiders......Page 283
Final Thoughts......Page 284
28: Keeping Webbots out of Trouble......Page 285
It’s All About Respect......Page 286
Don’t Be an Armchair Lawyer......Page 287
Trespass to Chattels......Page 290
Internet Law......Page 292
Final Thoughts......Page 293
Creating a Minimal PHP/CURL Session......Page 295
Setting PHP/CURL Options......Page 296
CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS......Page 297
CURLOPT_TIMEOUT......Page 298
CURLOPT_SSL_VERIFYPEER......Page 299
CURLOPT_VERBOSE......Page 300
Executing the PHP/CURL Command......Page 301
Viewing PHP/CURL Errors......Page 302
Closing PHP/CURL Sessions......Page 303
HTTP Codes......Page 305
NNTP Codes......Page 307
C: SMS Email Addresses......Page 309
Index......Page 312