Grokking Deep Learning

Contents......Page 3
Preface......Page 11
About this Book......Page 12
Welcome to......Page 15
Why you should learn deep learning......Page 16
Why you should read this book......Page 17
What you need to get started......Page 19
Summary......Page 20
Fundamental Concepts......Page 21
What is deep learning?......Page 22
What is machine learning?......Page 23
Supervised machine learning......Page 24
Unsupervised machine learning......Page 25
Parametric vs. nonparametric learning......Page 26
Supervised parametric learning......Page 27
Unsupervised parametric learning......Page 29
Nonparametric learning......Page 30
Summary......Page 31
Forward Propagation......Page 32
Step 1: Predict......Page 33
A simple neural network making a prediction......Page 35
What is a neural network?......Page 36
What does this neural network do?......Page 37
Making a prediction with multiple inputs......Page 39
Multiple inputs: What does this neural network do?......Page 41
Multiple inputs: Complete runnable code......Page 46
Making a prediction with multiple outputs......Page 47
Predicting with multiple inputs and outputs......Page 49
Multiple inputs and outputs: How does it work?......Page 51
Predicting on predictions......Page 53
A quick primer on NumPy......Page 55
Summary......Page 57
Gradient Descent......Page 58
Compare......Page 59
Learn......Page 60
Compare: Does your network make   good predictions?......Page 61
Why measure error?......Page 62
What’s the simplest form of neural learning?......Page 63
Hot and cold learning......Page 65
Characteristics of hot and cold learning......Page 66
Calculating both direction and amount from error......Page 67
One iteration of gradient descent......Page 69
Learning is just reducing error......Page 71
Let’s watch several steps of learning......Page 73
Why does this work? What is weight_delta, really?......Page 75
Tunnel vision on one concept......Page 77
A box with rods poking out of it......Page 78
Derivatives: Take two......Page 79
What you don’t really need to know......Page 80
How to use a derivative to learn......Page 81
Look familiar?......Page 82
Breaking gradient descent......Page 83
Visualizing the overcorrections......Page 84
Divergence......Page 85
Introducing alpha......Page 86
Alpha in code......Page 87
Memorizing......Page 88
Generalizing Gradient Descent......Page 89
Gradient descent learning with multiple inputs......Page 90
Gradient descent with multiple inputs explained......Page 92
Let’s watch several steps of learning......Page 96
Freezing one weight: What does it do?......Page 98
Gradient descent learning with multiple outputs......Page 100
Gradient descent with multiple inputs and outputs......Page 102
What do these weights learn?......Page 104
Visualizing weight values......Page 106
Visualizing dot products (weighted sums)......Page 107
Summary......Page 108
Intro to Backpropagation......Page 109
The streetlight problem......Page 110
Preparing the data......Page 112
Matrices and the matrix relationship......Page 113
Creating a matrix or two in Python......Page 116
Building a neural network......Page 117
Learning the whole dataset......Page 118
Full, batch, and stochastic gradient descent......Page 119
Neural networks learn correlation......Page 120
Up and down pressure......Page 121
Edge case: Overfittin......Page 123
Edge case: Confliting pressure......Page 124
Learning indirect correlation......Page 126
Creating correlation......Page 127
Stacking neural networks: A review......Page 128
Backpropagation: Long-distance error attribution......Page 129
Backpropagation: Why does this work?......Page 130
Linear vs. nonlinear......Page 131
Why the neural network still doesn’t work......Page 132
The secret to sometimes correlation......Page 133
A quick break......Page 134
Your first deep neual network......Page 135
Backpropagation in code......Page 136
One iteration of backpropagation......Page 138
Putting it all together......Page 140
Why do deep networks matter?......Page 141
Picture NNs......Page 142
It’s time to simplify......Page 143
Correlation summarization......Page 144
The previously overcomplicated visualization......Page 145
The simplified visualiztion......Page 146
Simplifying even further......Page 147
Let’s see this network predict......Page 148
Visualizing using letters instead of pictures......Page 149
Linking the variables......Page 150
Everything side by side......Page 151
The importance of visualization tools......Page 152
Intro to Regularization & Batching......Page 153
Three-layer network on MNIST......Page 154
Well, that was easy......Page 156
Memorization vs. generalization......Page 157
Overfitting in neual networks......Page 158
Where overfitting omes from......Page 159
The simplest regularization: Early stopping......Page 160
Industry standard regularization: Dropout......Page 161
Why dropout works: Ensembling works......Page 162
Dropout in code......Page 163
Dropout evaluated on MNIST......Page 165
Batch gradient descent......Page 166
Summary......Page 168
Activation Functions......Page 169
What is an activation function?......Page 170
Standard hidden-layer activation functions......Page 173
Standard output layer activation functions......Page 174
The core issue: Inputs have similarity......Page 176
softmax computation......Page 177
Activation installation instructions......Page 178
Multiplying delta by the slope......Page 180
Converting output to slope (derivative)......Page 181
Upgrading the MNIST network......Page 182
Intro to Convolutional NNs......Page 184
Reusing weights in multiple places......Page 185
The convolutional layer......Page 186
A simple implementation in NumPy......Page 188
Summary......Page 192
NNs that understand Language......Page 193
What does it mean to understand language?......Page 194
Natural language processing (NLP)......Page 195
Supervised NLP......Page 196
IMDB movie reviews dataset......Page 197
Capturing word correlation in input data......Page 198
Predicting movie reviews......Page 199
Intro to an embedding layer......Page 200
Interpreting the output......Page 202
Neural architecture......Page 203
Comparing word embeddings......Page 205
What is the meaning of a neuron?......Page 206
Filling in the blank......Page 207
Meaning is derived from loss......Page 209
King – Man + Woman ~= Queen......Page 212
Word analogies......Page 213
Summary......Page 214
Recurrent Layers for Variable-Length Data......Page 215
The challenge of arbitrary length......Page 216
Do comparisons really matter?......Page 217
The surprising power of averaged word vectors......Page 218
How is information stored in these embeddings?......Page 219
How does a neural network use embeddings?......Page 220
The limitations of bag-of-words vectors......Page 221
Using identity vectors to sum word embeddings......Page 222
Matrices that change absolutely nothing......Page 223
Learning the transition matrices......Page 224
Learning to create useful sentence vectors......Page 225
Forward propagation in Python......Page 226
How do you backpropagate into this?......Page 227
Let’s train it!......Page 228
Setting things up......Page 229
Forward propagation with arbitrary length......Page 230
Backpropagation with arbitrary length......Page 231
Weight update with arbitrary length......Page 232
Execution and output analysis......Page 233
Summary......Page 235
Deep Learning Framework......Page 236
What is a deep learning framework?......Page 237
Introduction to tensors......Page 238
Introduction to automatic gradient computation  (autograd)......Page 239
A quick checkpoint......Page 241
Tensors that are used multiple times......Page 242
Upgrading autograd to support multiuse tensors......Page 243
How does addition backpropagation work?......Page 245
Adding support for negation......Page 246
Adding support for additional functions......Page 247
Using autograd to train a neural network......Page 251
Adding automatic optimization......Page 253
Adding support for layer types......Page 254
Layers that contain layers......Page 255
Loss-function layers......Page 256
How to learn a framework......Page 257
Nonlinearity layers......Page 258
The embedding layer......Page 260
Adding indexing to autograd......Page 261
The embedding layer (revisited)......Page 262
The cross-entropy layer......Page 263
The recurrent neural network layer......Page 265
Summary......Page 268
Long Short-Term Memory......Page 269
Character language modeling......Page 270
The need for truncated backpropagation......Page 271
Truncated backpropagation......Page 272
A sample of the output......Page 275
Vanishing and exploding gradients......Page 276
A toy example of RNN backpropagation......Page 277
Long short-term memory (LSTM) cells......Page 278
Some intuition about LSTM gates......Page 279
The long short-term memory layer......Page 280
Upgrading the character language model......Page 281
Training the LSTM character language model......Page 282
Tuning the LSTM character language model......Page 283
Summary......Page 284
Federated Learning......Page 285
The problem of privacy in deep learning......Page 286
Federated learning......Page 287
Learning to detect spam......Page 288
Let’s make it federated......Page 290
Hacking into federated learning......Page 291
Secure aggregation......Page 292
Homomorphic encryption......Page 293
Homomorphically encrypted federated learning......Page 294
Summary......Page 295
Where to go from here......Page 296
Index......Page 303