Machine Learning in Action

Machine Learning in Action......Page 1
brief contents......Page 8
contents......Page 10
preface......Page 18
acknowledgments......Page 20
Audience......Page 22
Part 2 Forecasting numeric values with regression......Page 23
Code conventions and downloads......Page 24
Author Online......Page 25
about the author......Page 26
about the cover illustration......Page 27
Classification......Page 28
Machine learning basics......Page 30
1.1 What is machine learning?......Page 32
1.1.1 Sensors and the data deluge......Page 33
1.2 Key terminology......Page 34
1.3 Key tasks of machine learning......Page 37
1.5 Steps in developing a machine learning application......Page 38
1.6.2 Python is popular......Page 40
1.6.4 Drawbacks......Page 41
1.7 Getting started with the NumPy library......Page 42
1.8 Summary......Page 44
Classifying with k-Nearest Neighbors......Page 45
2.1 Classifying with distance measurements......Page 46
2.1.1 Prepare: importing data with Python......Page 48
2.1.2 Putting the kNN classification algorithm into action......Page 50
2.2 Example: improving matches from a dating site with kNN......Page 51
2.2.1 Prepare: parsing data from a text file......Page 52
2.2.2 Analyze: creating scatter plots with Matplotlib......Page 54
2.2.3 Prepare: normalizing numeric values......Page 56
2.2.4 Test: testing the classifier as a whole program......Page 58
2.2.5 Use: putting together a useful system......Page 59
2.3.1 Prepare: converting images into test vectors......Page 60
2.3.2 Test: kNN on handwritten digits......Page 62
2.4 Summary......Page 63
Splitting datasets one feature at a time: decision trees......Page 64
3.1 Tree construction......Page 66
3.1.1 Information gain......Page 67
3.1.2 Splitting the dataset......Page 70
3.1.3 Recursively building the tree......Page 73
3.2 Plotting trees in Python with Matplotlib annotations......Page 75
3.2.1 Matplotlib annotations......Page 76
3.2.2 Constructing a tree of annotations......Page 78
3.3.1 Test: using the tree for classification......Page 83
3.4 Example: using decision trees to predict contact lens type......Page 84
3.5 Summary......Page 86
Classifying with probability theory: naïve Bayes......Page 88
4.1 Classifying with Bayesian decision theory......Page 89
4.2 Conditional probability......Page 90
4.4 Document classification with naïve Bayes......Page 92
4.5.1 Prepare: making word vectors from text......Page 94
4.5.2 Train: calculating probabilities from word vectors......Page 96
4.5.3 Test: modifying the classifier for real-world conditions......Page 98
4.5.4 Prepare: the bag-of-words document model......Page 100
4.6.1 Prepare: tokenizing text......Page 101
4.6.2 Test: cross validation with naïve Bayes......Page 102
4.7 Example: using naïve Bayes to reveal local attitudes from personal ads......Page 104
4.7.1 Collect: importing RSS feeds......Page 105
4.7.2 Analyze: displaying locally used words......Page 107
4.8 Summary......Page 109
Logistic regression......Page 110
5.1 Classification with logistic regression and the sigmoid function: a tractable step function......Page 111
5.2.1 Gradient ascent......Page 113
5.2.2 Train: using gradient ascent to find the best parameters......Page 115
5.2.3 Analyze: plotting the decision boundary......Page 117
5.2.4 Train: stochastic gradient ascent......Page 118
5.3 Example: estimating horse fatalities from colic......Page 123
5.3.1 Prepare: dealing with missing values in the data......Page 124
5.3.2 Test: classifying with logistic regression......Page 125
5.4 Summary......Page 127
Support vector machines......Page 128
6.1 Separating data with the maximum margin......Page 129
6.2.1 Framing the optimization problem in terms of our classifier......Page 131
6.3.1 Platt’s SMO algorithm......Page 133
6.3.2 Solving small datasets with the simplified SMO......Page 134
6.4 Speeding up optimization with the full Platt SMO......Page 139
6.5.1 Mapping data to higher dimensions with kernels......Page 145
6.5.2 The radial bias function as a kernel......Page 146
6.5.3 Using a kernel for testing......Page 149
6.6 Example: revisiting handwriting classification......Page 152
6.7 Summary......Page 154
Improving classification with the AdaBoost meta-algorithm......Page 156
7.1.1 Building classifiers from randomly resampled data: bagging......Page 157
7.2 Train: improving the classifier by focusing on errors......Page 158
7.3 Creating a weak learner with a decision stump......Page 160
7.4 Implementing the full AdaBoost algorithm......Page 163
7.5 Test: classifying with AdaBoost......Page 166
7.6 Example: AdaBoost on a difficult dataset......Page 167
7.7 Classification imbalance......Page 169
7.7.1 Alternative performance metrics: precision, recall, and ROC......Page 170
7.7.2 Manipulating the classifier’s decision with a cost function......Page 174
7.8 Summary......Page 175
Forecasting numeric values with regression......Page 178
Predicting numeric values: regression......Page 180
8.1 Finding best-fit lines with linear regression......Page 181
8.2 Locally weighted linear regression......Page 187
8.3 Example: predicting the age of an abalone......Page 190
8.4.1 Ridge regression......Page 191
8.4.3 Forward stagewise regression......Page 194
8.5 The bias/variance tradeoff......Page 197
8.6 Example: forecasting the price of LEGO sets......Page 199
8.6.1 Collect: using the Google shopping API......Page 200
8.6.2 Train: building a model......Page 201
8.7 Summary......Page 204
Tree-based regression......Page 206
9.1 Locally modeling complex data......Page 207
9.2 Building trees with continuous and discrete features......Page 208
9.3.1 Building the tree......Page 211
9.3.2 Executing the code......Page 213
9.4.1 Prepruning......Page 215
9.4.2 Postpruning......Page 217
9.5 Model trees......Page 219
9.6 Example: comparing tree methods to standard regression......Page 222
9.7 Using Tkinter to create a GUI in Python......Page 225
9.7.1 Building a GUI in Tkinter......Page 226
9.7.2 Interfacing Matplotlib and Tkinter......Page 228
9.8 Summary......Page 230
Unsupervised learning......Page 232
Grouping unlabeled items using k-means clustering......Page 234
10.1 The k-means clustering algorithm......Page 235
10.2 Improving cluster performance with postprocessing......Page 240
10.3 Bisecting k-means......Page 241
10.4 Example: clustering points on a map......Page 244
10.4.1 The Yahoo! PlaceFinder API......Page 245
10.4.2 Clustering geographic coordinates......Page 247
10.5 Summary......Page 250
Association analysis with the Apriori algorithm......Page 251
11.1 Association analysis......Page 252
11.2 The Apriori principle......Page 253
11.3 Finding frequent itemsets with the Apriori algorithm......Page 255
11.3.1 Generating candidate itemsets......Page 256
11.3.2 Putting together the full Apriori algorithm......Page 258
11.4 Mining association rules from frequent item sets......Page 260
11.5 Example: uncovering patterns in congressional voting......Page 264
11.5.1 Collect: build a transaction data set of congressional voting records......Page 265
11.5.2 Test: association rules from congressional voting records......Page 270
11.6 Example: finding similar features in poisonous mushrooms......Page 272
11.7 Summary......Page 273
Efficiently finding frequent itemsets with FP-growth......Page 275
12.1 FP-trees: an efficient way to encode a dataset......Page 276
12.2.1 Creating the FP-tree data structure......Page 278
12.2.2 Constructing the FP-tree......Page 279
12.3 Mining frequent items from an FP-tree......Page 283
12.3.1 Extracting conditional pattern bases......Page 284
12.3.2 Creating conditional FP-trees......Page 285
12.4 Example: finding co-occurring words in a Twitter feed......Page 287
12.5 Example: mining a clickstream from a news site......Page 291
12.6 Summary......Page 292
Additional tools......Page 294
Using principal component analysis to simplify data......Page 296
13.1 Dimensionality reduction techniques......Page 297
13.2.1 Moving the coordinate axes......Page 298
13.2.2 Performing PCA in NumPy......Page 300
13.3 Example: using PCA to reduce the dimensionality of semiconductor manufacturing data......Page 302
13.4 Summary......Page 305
Simplifying data with the singular value decomposition......Page 307
14.1.1 Latent semantic indexing......Page 308
14.1.2 Recommendation systems......Page 309
14.2 Matrix factorization......Page 310
14.3 SVD in Python......Page 311
14.4 Collaborative filtering–based recommendation engines......Page 313
14.4.1 Measuring similarity......Page 314
14.4.3 Evaluating recommendation engines......Page 316
14.5.1 Recommending untasted dishes......Page 317
14.5.2 Improving recommendations with the SVD......Page 319
14.6 Example: image compression with the SVD......Page 322
14.7 Summary......Page 325
Big data and MapReduce......Page 326
15.1 MapReduce: a framework for distributed computing......Page 327
15.2 Hadoop Streaming......Page 329
15.2.1 Distributed mean and variance mapper......Page 330
15.2.2 Distributed mean and variance reducer......Page 331
15.3.1 Services available on AWS......Page 332
15.3.2 Getting started with Amazon Web Services......Page 333
15.3.3 Running a Hadoop job on EMR......Page 334
15.4 Machine learning in MapReduce......Page 339
15.5.1 Using mrjob for seamless integration with EMR......Page 340
15.5.2 The anatomy of a MapReduce script in mrjob......Page 341
15.6 Example: the Pegasos algorithm for distributed SVMs......Page 343
15.6.1 The Pegasos algorithm......Page 344
15.6.2 Training: MapReduce support vector machines with mrjob......Page 345
15.7 Do you really need MapReduce?......Page 349
15.8 Summary......Page 350
A.1.1 Windows......Page 352
A.2 A quick introduction to Python......Page 353
A.2.1 Collection types......Page 354
A.2.2 Control structures......Page 355
A.2.3 List comprehensions......Page 356
A.3 A quick introduction to NumPy......Page 357
A.5 Mrjob......Page 360
A.7 Python-Twitter......Page 361
B.1 Matrices......Page 362
B.2 Matrix inverse......Page 365
B.3 Norms......Page 366
B.4 Matrix calculus......Page 367
C.1 Intro to probability......Page 368
C.2 Joint probability......Page 369
C.3 Basic rules of probability......Page 370
appendix D: Resources......Page 372
C......Page 374
D......Page 375
I......Page 376
M......Page 377
P......Page 378
S......Page 379
T......Page 380
Y......Page 381