Collective Intelligence in Action

brief contents......Page 6
contents......Page 8
foreword......Page 16
preface......Page 18
acknowledgments......Page 20
about this book......Page 22
Roadmap......Page 23
About the author......Page 25
About the cover illustration......Page 26
Gathering data for intelligence......Page 28
Understanding collective intelligence......Page 30
1.1 What is collective intelligence?......Page 31
1.2 CI in web applications......Page 33
1.2.1 Collective intelligence from the ground up: a sample application......Page 34
1.2.2 Benefits of collective intelligence......Page 36
1.2.3 CI is the core component of Web 2.0......Page 37
1.2.4 Harnessing CI to transform from content-centric to user-centric applications......Page 39
1.3.1 Explicit intelligence......Page 41
1.3.2 Implicit intelligence......Page 42
1.3.3 Derived intelligence......Page 43
1.5 Resources......Page 45
Learning from user interactions......Page 47
2.1.1 Synchronous and asynchronous services......Page 48
2.1.2 Real-time learning in an event-driven system......Page 50
2.1.3 Polling services for non–event-driven systems......Page 51
2.2 Basics of algorithms for applying CI......Page 52
2.2.1 Users and items......Page 53
2.2.2 Representing user information......Page 54
2.2.3 Content-based analysis and collaborative filtering......Page 56
2.2.4 Representing intelligence from unstructured text......Page 57
2.2.5 Computing similarities......Page 58
2.2.6 Types of datasets......Page 59
2.3 Forms of user interaction......Page 61
2.3.1 Rating and voting......Page 62
2.3.3 Bookmarking and saving......Page 63
2.3.5 Click-stream......Page 64
2.3.6 Reviews......Page 66
2.4.1 Intelligence from ratings via an example......Page 68
2.4.2 Intelligence from bookmarking, saving, purchasing Items, forwarding, click-stream, and reviews......Page 73
2.6 Resources......Page 75
Extracting intelligence from tags......Page 77
3.1 Introduction to tagging......Page 78
3.1.2 Professionally generated tags......Page 79
3.1.3 User-generated tags......Page 80
3.1.4 Machine-generated tags......Page 81
3.1.6 Why do users tag?......Page 82
3.2.1 Building dynamic navigation......Page 83
3.2.2 Innovative uses of tag clouds......Page 85
3.2.3 Targeted search......Page 86
3.3 Extracting intelligence from user tagging: an example......Page 87
3.3.2 Items of interest for a user......Page 88
3.4 Scalable persistence architecture for tagging......Page 89
3.4.1 Reviewing other approaches......Page 90
3.4.2 Recommended persistence architecture......Page 93
3.5.1 Persistence design for tag clouds......Page 96
3.5.2 Algorithm for building a tag cloud......Page 97
3.5.3 Implementing a tag cloud......Page 98
3.5.4 Visualizing a tag cloud......Page 103
3.6 Finding similar tags......Page 106
3.7 Summary......Page 107
3.8 Resources......Page 108
Extracting intelligence from content......Page 109
4.1.1 Classifying content......Page 110
4.1.2 Architecture for integrating content......Page 112
4.2 The main CI-related content types......Page 113
4.2.1 Blogs......Page 114
4.2.2 Wikis......Page 116
4.2.3 Groups and message boards......Page 118
4.3 Extracting intelligence step by step......Page 120
4.3.1 Setting up the example......Page 121
4.3.2 Naïve analysis......Page 122
4.3.3 Removing common words......Page 125
4.3.4 Stemming......Page 126
4.3.5 Detecting phrases......Page 127
4.4 Simple and composite content types......Page 129
4.5 Summary......Page 130
4.6 Resources......Page 131
Searching the blogosphere......Page 134
5.1.1 Leveraging the blogosphere......Page 135
5.1.2 RSS: the publishing format......Page 136
5.2 Building a framework to search the blogosphere......Page 138
5.2.2 The search parameters......Page 140
5.2.3 The query results......Page 141
5.2.4 Handling the XML response......Page 142
5.3 Implementing the base classes......Page 143
5.3.2 Implementing the result objects......Page 144
5.3.3 Implementing the searcher......Page 146
5.3.4 Parsing XML response......Page 150
5.3.5 Extending the framework......Page 154
5.4.1 Technorati search API overview......Page 155
5.4.2 Implementing classes for integrating Technorati......Page 157
5.5.1 Bloglines search API overview......Page 162
5.5.2 Implementing classes for integrating Bloglines......Page 163
5.6.1 Generalizing the query parameters......Page 166
5.6.2 Generalizing the blog searcher......Page 167
5.6.3 Building the RSS 2.0 XML parser......Page 168
5.8 Resources......Page 170
Intelligent web crawling......Page 172
6.1.1 Why crawl the Web?......Page 173
6.1.2 The crawling process......Page 174
6.1.3 Intelligent crawling and focused crawling......Page 176
6.1.4 Deep crawling......Page 177
6.1.5 Available crawlers......Page 178
6.2.1 Implementing the core algorithm......Page 179
6.2.2 Being polite: following the robots.txt file......Page 183
6.2.3 Retrieving the content......Page 186
6.2.4 Extracting URLs......Page 187
6.2.5 Making the crawler intelligent......Page 188
6.2.6 Running the crawler......Page 189
6.2.7 Extending the crawler......Page 190
6.3.1 Setting up Nutch......Page 191
6.3.2 Running the Nutch crawler......Page 192
6.3.3 Searching with Nutch......Page 195
6.3.4 Apache Hadoop, MapReduce, and Dryad......Page 196
6.5 Resources......Page 198
Deriving intelligence......Page 200
Data mining: process, toolkits, and standards......Page 202
7.1.1 Attributes......Page 203
7.1.3 Key learning algorithms......Page 205
7.1.4 The mining process......Page 208
7.2 Using an open source data mining framework: WEKA......Page 209
7.2.1 Using the WEKA application: a step-by-step tutorial......Page 210
7.2.2 Understanding the WEKA APIs......Page 213
7.2.3 Using the WEKA APIs via an example......Page 215
7.3 Standard data mining API: Java Data Mining (JDM)......Page 220
7.3.1 JDM architecture......Page 221
7.3.2 Key JDM objects......Page 222
7.3.3 Representing the dataset......Page 223
7.3.4 Learning models......Page 224
7.3.6 JDM tasks......Page 226
7.3.7 JDM connection......Page 227
7.3.8 Sample code for accessing DME......Page 229
7.4 Summary......Page 231
7.5 Resources......Page 232
Building a text analysis toolkit......Page 233
8.1 Building the text analyzers......Page 234
8.1.1 Leveraging Lucene......Page 235
8.1.2 Writing a stemmer analyzer......Page 240
8.1.3 Writing a TokenFilter to inject synonyms and detect phrases......Page 241
8.1.5 Putting our analyzers to work......Page 245
8.2 Building the text analysis infrastructure......Page 248
8.2.1 Building the tag infrastructure......Page 249
8.2.2 Building the term vector infrastructure......Page 252
8.2.3 Building the Text Analyzer class......Page 258
8.2.4 Applying the text analysis infrastructure......Page 261
8.3 Use cases for applying the framework......Page 264
8.4 Summary......Page 265
8.5 Resources......Page 266
Discovering patterns with clustering......Page 267
9.1 Clustering blog entries......Page 268
9.1.1 Defining the text clustering infrastructure......Page 269
9.1.2 Retrieving blog entries from Technorati......Page 271
9.1.3 Implementing the k-means algorithms for text processing......Page 274
9.1.4 Implementing hierarchical clustering algorithms for text processing......Page 280
9.1.5 Expectation maximization and other examples of clustering high-dimension sparse data......Page 288
9.2 Leveraging WEKA for clustering......Page 289
9.2.1 Creating the learning dataset......Page 290
9.2.2 Creating the clusterer......Page 292
9.2.3 Evaluating the clustering results......Page 293
9.3.1 Key JDM clustering-related classes......Page 295
9.3.2 Clustering settings using the JDM APIs......Page 296
9.3.4 Executing the clustering task using the JDM APIs......Page 298
9.4 Summary......Page 299
9.5 Resources......Page 300
Making predictions......Page 301
10.1.1 Learning decision trees by example......Page 302
10.1.2 Naïve Bayes’ classifier......Page 308
10.1.3 Belief networks......Page 312
10.2 Classifying blog entries using WEKA APIs......Page 314
10.2.1 Building the dataset for classifying blog entries......Page 315
10.2.2 Building the classifier class......Page 319
10.3 Regression fundamentals......Page 321
10.3.1 Linear regression......Page 322
10.3.2 Multi-layer perceptron (MLP)......Page 324
10.3.3 Radial basis functions (RBF)......Page 325
10.4 Regression using WEKA......Page 326
10.5.1 Key JDM supervised learning–related classes......Page 327
10.5.2 Supervised learning settings using the JDM APIs......Page 329
10.5.4 Executing the classification task using the JDM APIs......Page 331
10.5.6 Retrieving the classification model using the JDM APIs......Page 332
10.7 Resources......Page 333
Applying intelligence in your application......Page 334
Intelligent search......Page 336
11.1.1 Search architecture......Page 337
11.1.2 Core Lucene classes......Page 338
11.1.3 Basic indexing and searching via example......Page 340
11.2.1 Understanding the index format......Page 347
11.2.2 Modifying the index......Page 348
11.2.3 Incremental indexing......Page 349
11.2.4 Accessing the term frequency vector......Page 351
11.2.5 Optimizing indexing performance......Page 352
11.3.1 Understanding Lucene scoring......Page 354
11.3.2 Querying Lucene......Page 357
11.3.3 Sorting search results......Page 358
11.3.4 Querying on multiple fields......Page 360
11.3.5 Filtering......Page 361
11.3.7 Using a HitCollector......Page 362
11.3.8 Optimizing search performance......Page 365
11.4.2 Solr......Page 366
11.5 Approaches to intelligent search......Page 368
11.5.2 Clustering search results......Page 369
11.5.4 Community-based search......Page 371
11.5.6 Data search......Page 372
11.7 Resources......Page 374
Building a recommendation engine......Page 376
12.1 Recommendation engine fundamentals......Page 377
12.1.1 Introducing the recommendation engine......Page 378
12.1.2 Item-based and user-based analysis......Page 379
12.1.3 Computing similarity using content-based and collaborative techniques......Page 380
12.1.4 Comparison of content-based and collaborative techniques......Page 381
12.2.1 Finding similar items using a search engine (Lucene)......Page 382
12.2.2 Building a content-based recommendation engine......Page 386
12.2.4 Personalizing content for a user......Page 389
12.3.1 k-nearest neighbor......Page 390
12.3.2 Packages for implementing collaborative filtering......Page 392
12.3.3 Dimensionality reduction with latent semantic indexing......Page 396
12.3.4 Implementing dimensionality reduction......Page 397
12.4 Real-world solutions......Page 400
12.4.1 Amazon item-to-item recommendation......Page 401
12.4.2 Google News personalization......Page 404
12.4.3 Netflix and the BellKor Solution for the Netflix Prize......Page 408
12.5 Summary......Page 412
12.6 Resources......Page 413
B......Page 416
C......Page 417
F......Page 418
I......Page 419
M......Page 420
P......Page 421
S......Page 422
T......Page 423
Z......Page 424