Python Machine Learning. Machine Learning and Deep Learning with Python, scikit-learn and TensorFlow

Cover......Page 1
Copyright......Page 3
Credits......Page 4
About the Authors......Page 5
About the Reviewers......Page 7
www.PacktPub.com......Page 8
Customer Feedback......Page 9
Table of Contents......Page 10
Preface......Page 20
Chapter 1: Giving Computers the Ability to Learn from Data......Page 28
 The three different types of machine learning......Page 29
   Classification for predicting class labels......Page 30
   Regression for predicting continuous outcomes......Page 32
  Solving interactive problems with reinforcement learning......Page 33
   Finding subgroups with clustering......Page 34
 Introduction to the basic terminology and notations......Page 35
 A roadmap for building machine learning systems......Page 38
  Training and selecting a predictive model......Page 39
 Using Python for machine learning......Page 40
  Using the Anaconda Python distribution and package manager......Page 41
 Summary......Page 42
Chapter 2: Training Simple Machine Learning Algorithms for Classification......Page 44
 Artificial neurons – a brief glimpse into the early history of machine learning......Page 45
  The formal definition of an artificial neuron......Page 46
  The perceptron learning rule......Page 48
  An object-oriented perceptron API......Page 51
  Training a perceptron model on the Iris dataset......Page 55
 Adaptive linear neurons and the convergence of learning......Page 61
  Minimizing cost functions with gradient descent......Page 62
  Implementing Adaline in Python......Page 65
  Improving gradient descent through feature scaling......Page 69
  Large-scale machine learning and stochastic gradient descent......Page 71
 Summary......Page 77
Chapter 3: A Tour of Machine Learning Classifiers Using scikit-learn......Page 78
 First steps with scikit-learn – training a perceptron......Page 79
  Logistic regression intuition and conditional probabilities......Page 86
  Learning the weights of the logistic cost function......Page 90
  Converting an Adaline implementation into an algorithm for logistic regression......Page 93
  Training a logistic regression model with scikit-learn......Page 98
  Tackling overfitting via regularization......Page 100
 Maximum margin classification with support vector machines......Page 103
  Maximum margin intuition......Page 104
  Dealing with a nonlinearly separable case using slack variables......Page 106
  Alternative implementations in scikit-learn......Page 108
  Kernel methods for linearly inseparable data......Page 109
  Using the kernel trick to find separating hyperplanes in high-dimensional space......Page 111
 Decision tree learning......Page 115
  Maximizing information gain – getting the most bang for your buck......Page 117
  Building a decision tree......Page 122
  Combining multiple decision trees via random forests......Page 125
 K-nearest neighbors – a lazy learning algorithm......Page 128
 Summary......Page 132
 Dealing with missing data......Page 134
  Identifying missing values in tabular data......Page 135
  Eliminating samples or features with missing values......Page 136
  Imputing missing values......Page 137
  Understanding the scikit-learn estimator API......Page 138
 Handling categorical data......Page 139
  Mapping ordinal features......Page 140
  Encoding class labels......Page 141
  Performing one-hot encoding on nominal features......Page 143
 Partitioning a dataset into separate training and test sets......Page 145
 Bringing features onto the same scale......Page 147
 Selecting meaningful features......Page 150
  A geometric interpretation of L2 regularization......Page 151
  Sparse solutions with L1 regularization......Page 153
  Sequential feature selection algorithms......Page 157
 Assessing feature importance with random forests......Page 163
 Summary......Page 166
Chapter 5: Compressing Data via Dimensionality Reduction......Page 168
  The main steps behind principal component analysis......Page 169
  Extracting the principal components step by step......Page 171
  Total and explained variance......Page 174
  Feature transformation......Page 175
  Principal component analysis in scikit-learn......Page 178
  Principal component analysis versus linear discriminant analysis......Page 182
  The inner workings of linear discriminant analysis......Page 183
  Computing the scatter matrices......Page 184
  Selecting linear discriminants for the new feature subspace......Page 187
  Projecting samples onto the new feature space......Page 189
  LDA via scikit-learn......Page 190
 Using kernel principal component analysis for nonlinear mappings......Page 192
  Kernel functions and the kernel trick......Page 193
  Implementing a kernel principal component analysis in Python......Page 199
   Example 1 – separating half-moon shapes......Page 200
   Example 2 – separating concentric circles......Page 203
  Projecting new data points......Page 206
  Kernel principal component analysis in scikit-learn......Page 210
 Summary......Page 211
 Streamlining workflows with pipelines......Page 212
  Loading the Breast Cancer Wisconsin dataset......Page 213
  Combining transformers and estimators in a pipeline......Page 214
 Using k-fold cross-validation to assess model performance......Page 216
  The holdout method......Page 217
  K-fold cross-validation......Page 218
 Debugging algorithms with learning and validation curves......Page 222
  Diagnosing bias and variance problems with learning curves......Page 223
  Addressing over- and underfitting with validation curves......Page 226
  Tuning hyperparameters via grid search......Page 228
  Algorithm selection with nested cross-validation......Page 230
 Looking at different performance evaluation metrics......Page 232
  Reading a confusion matrix......Page 233
  Optimizing the precision and recall of a classification model......Page 234
  Plotting a receiver operating characteristic......Page 237
  Scoring metrics for multiclass classification......Page 240
 Dealing with class imbalance......Page 241
 Summary......Page 243
 Learning with ensembles......Page 246
  Implementing a simple majority vote classifier......Page 251
  Using the majority voting principle to make predictions......Page 258
  Evaluating and tuning the ensemble classifier......Page 261
  Bagging in a nutshell......Page 267
  Applying bagging to classify samples in the Wine dataset......Page 269
  How boosting works......Page 273
  Applying AdaBoost using scikit-learn......Page 278
 Summary......Page 281
Chapter 8: Applying Machine Learning to Sentiment Analysis......Page 282
  Obtaining the movie review dataset......Page 283
  Preprocessing the movie dataset into more convenient format......Page 284
  Transforming words into feature vectors......Page 286
  Assessing word relevancy via term frequency-inverse document frequency......Page 288
  Cleaning text data......Page 291
  Processing documents into tokens......Page 293
 Training a logistic regression model for document classification......Page 295
 Working with bigger data – online algorithms and out-of-core learning......Page 297
 Topic modeling with Latent Dirichlet Allocation......Page 301
  LDA with scikit-learn......Page 302
 Summary......Page 306
Chapter 9: Embedding a Machine Learning Model into a Web Application......Page 308
 Serializing fitted scikit-learn estimators......Page 309
 Setting up an SQLite database for data storage......Page 312
 Developing a web application with Flask......Page 314
  Our first Flask web application......Page 315
  Form validation and rendering......Page 317
   Setting up the directory structure......Page 318
   Implementing a macro using the Jinja2 templating engine......Page 319
   Adding style via CSS......Page 320
 Turning the movie review classifier into a web application......Page 321
  Files and folders – looking at the directory tree......Page 323
  Implementing the main application as app.py......Page 325
  Setting up the review form......Page 327
  Creating a results page template......Page 329
  Creating a PythonAnywhere account......Page 331
  Uploading the movie classifier application......Page 332
  Updating the movie classifier......Page 333
 Summary......Page 335
Chapter 10: Predicting Continuous Target Variables with Regression Analysis......Page 336
  Simple linear regression......Page 337
  Multiple linear regression......Page 338
 Exploring the Housing dataset......Page 339
  Loading the Housing dataset into a data frame......Page 340
  Visualizing the important characteristics of a dataset......Page 341
  Looking at relationships using a correlation matrix......Page 343
  Solving regression for regression parameters with gradient descent......Page 346
  Estimating coefficient of a regression model via scikit-learn......Page 351
 Fitting a robust regression model using RANSAC......Page 352
 Evaluating the performance of linear regression models......Page 355
 Using regularized methods for regression......Page 359
  Adding polynomial terms using scikit-learn......Page 361
  Modeling nonlinear relationships in the Housing dataset......Page 363
 Dealing with nonlinear relationships using random forests......Page 366
  Decision tree regression......Page 367
  Random forest regression......Page 369
 Summary......Page 372
Chapter 11: Working with Unlabeled Data – Clustering Analysis......Page 374
  K-means clustering using scikit-learn......Page 375
  A smarter way of placing the initial cluster centroids using k-means++......Page 380
  Hard versus soft clustering......Page 381
  Using the elbow method to find the optimal number of clusters......Page 384
  Quantifying the quality of clustering via silhouette plots......Page 385
 Organizing clusters as a hierarchical tree......Page 390
  Grouping clusters in bottom-up fashion......Page 391
  Performing hierarchical clustering on a distance matrix......Page 392
  Attaching dendrograms to a heat map......Page 396
  Applying agglomerative clustering via scikit-learn......Page 398
 Locating regions of high density via DBSCAN......Page 399
 Summary......Page 405
Chapter 12: Implementing a Multilayer Artificial Neural Network from Scratch......Page 406
 Modeling complex functions with artificial neural networks......Page 407
  Single-layer neural network recap......Page 409
  Introducing the multilayer neural network architecture......Page 411
  Activating a neural network via forward propagation......Page 414
 Classifying handwritten digits......Page 416
  Obtaining the MNIST dataset......Page 417
  Implementing a multilayer perceptron......Page 423
 Training an artificial neural network......Page 434
  Computing the logistic cost function......Page 435
  Developing your intuition for backpropagation......Page 438
  Training neural networks via backpropagation......Page 439
 About the convergence in neural networks......Page 444
 A few last words about the neural network implementation......Page 445
 Summary......Page 446
 TensorFlow and training performance......Page 448
  What is TensorFlow?......Page 450
  First steps with TensorFlow......Page 451
  Working with array structures......Page 454
  Developing a simple model with the low-level TensorFlow API......Page 455
 Training neural networks efficiently with high-level TensorFlow APIs......Page 460
  Building multilayer neural networks using TensorFlow's Layers API......Page 461
  Developing a multilayer neural network with Keras......Page 465
 Choosing activation functions for multilayer networks......Page 470
  Logistic function recap......Page 471
  Estimating class probabilities in multiclass classification via the softmax function......Page 473
  Broadening the output spectrum using a hyperbolic tangent......Page 474
  Rectified linear unit activation......Page 476
 Summary......Page 478
Chapter 14: Going Deeper – The Mechanics of TensorFlow......Page 480
 TensorFlow ranks and tensors......Page 481
  How to get the rank and shape of a tensor......Page 482
 Understanding TensorFlow's computation graphs......Page 483
  Defining placeholders......Page 486
  Feeding placeholders with data......Page 487
  Defining placeholders for data arrays with varying batchsizes......Page 488
 Variables in TensorFlow......Page 489
  Defining variables......Page 490
  Initializing variables......Page 492
  Variable scope......Page 493
  Reusing variables......Page 495
 Building a regression model......Page 498
 Executing objects in a TensorFlow graph using their names......Page 502
 Saving and restoring a model in TensorFlow......Page 503
 Transforming Tensors as multidimensional data arrays......Page 506
 Utilizing control flow mechanics in building graphs......Page 510
 Visualizing the graph with TensorBoard......Page 514
  Extending your TensorBoard experience......Page 517
 Summary......Page 518
Chapter 15: Classifying Images with Deep Convolutional Neural Networks......Page 520
  Understanding CNNs and learning feature hierarchies......Page 521
   Performing a discrete convolution in one dimension......Page 523
   The effect of zero-padding in a convolution......Page 526
   Determining the size of the convolution output......Page 528
   Performing a discrete convolution in 2D......Page 529
  Subsampling......Page 533
  Working with multiple input or color channels......Page 535
  Regularizing a neural network with dropout......Page 539
  The multilayer CNN architecture......Page 541
  Loading and preprocessing the data......Page 543
  Implementing a CNN in the TensorFlow low-level API......Page 544
  Implementing a CNN in the TensorFlow layers API......Page 557
 Summary......Page 563
Chapter 16: Modeling Sequential Data Using Recurrent Neural Networks......Page 564
  Modeling sequential data – order matters......Page 565
  Representing sequences......Page 566
  The different categories of sequence modeling......Page 567
  Understanding the structure and flow of an RNN......Page 568
  Computing activations in an RNN......Page 570
  The challenges of learning long-range interactions......Page 573
  LSTM units......Page 575
 Implementing a multilayer RNN for sequence modeling in TensorFlow......Page 577
 Project one – performing sentiment analysis of IMDb movie reviews using multilayer RNNs......Page 578
  Preparing the data......Page 579
  Embedding......Page 583
  Building an RNN model......Page 585
  The SentimentRNN class constructor......Page 586
  The build method......Page 587
   Step 2 – defining the initial states for the RNN cells......Page 589
  The train method......Page 590
  Instantiating the SentimentRNN class......Page 592
  Training and optimizing the sentiment analysis RNN model......Page 593
 Project two – implementing an RNN for character-level language modeling in TensorFlow......Page 594
  Preparing the data......Page 595
  Building a character-level RNN model......Page 599
  The constructor......Page 600
  The build method......Page 601
  The train method......Page 603
  The sample method......Page 605
  Creating and training the CharRNN Model......Page 606
 Chapter and book summary......Page 607
Index......Page 610