Statistical Mechanics of Learning

Half-title......Page 3
Title......Page 5
Copyright......Page 6
Contents......Page 7
Preface......Page 11
1.1 Artificial neural networks......Page 15
1.2 A simple example......Page 18
1.3 General setup......Page 22
1.4 Problems......Page 27
2.1 Gibbs learning......Page 28
2.2 The annealed approximation......Page 32
2.3 The Gardner analysis......Page 36
2.4 Summary......Page 41
2.5 Problems......Page 43
3.1 The Hebb rule......Page 47
3.2 The perceptron rule......Page 50
3.3 The pseudo-inverse rule......Page 51
3.4 The adaline rule......Page 53
3.5 Maximal stability......Page 54
3.6 The Bayes rule......Page 56
3.8 Problems......Page 60
4.1 Maximal stabilities......Page 63
4.2 Gibbs learning at non-zero temperature......Page 66
4.3 General statistical mechanics formulation......Page 70
4.4 Learning rules revisited......Page 73
4.5 The optimal potential......Page 77
4.6 Summary......Page 78
4.7 Problems......Page 79
5.1 Sources of noise......Page 83
5.2 Trying perfect learning......Page 86
5.3 Learning with errors......Page 92
5.4 Refinements......Page 94
5.5 Summary......Page 96
5.6 Problems......Page 97
6.1 The storage capacity......Page 99
6.2 Counting dichotomies: the Cover analysis......Page 103
6.3 Galilean pastiche: the Ising perceptron......Page 107
6.4 The distribution of stabilities......Page 112
6.5 Beyond the storage capacity......Page 116
6.6 Problems......Page 118
7.1 Smooth networks......Page 123
7.2 The Ising perceptron......Page 125
7.3 The reversed wedge perceptron......Page 128
7.4 The dynamics of discontinuous learning......Page 132
7.5 Summary......Page 135
7.6 Problems......Page 136
8.1 Supervised versus unsupervised learning......Page 139
8.2 The deceptions of randomness......Page 143
8.3 Learning a symmetry-breaking direction......Page 147
8.4 Clustering through competitive learning......Page 151
8.5 Clustering by tuning the temperature......Page 156
8.6 Discussion......Page 159
8.7 Problems......Page 161
9.1 Stochastic gradient descent......Page 163
9.2 Specific examples......Page 166
9.3 Optimal on-line learning......Page 169
9.4 Perceptron with a smooth transfer function......Page 173
9.5 Queries......Page 174
9.6 Unsupervised on-line learning......Page 179
9.7 The natural gradient......Page 183
9.8 Discussion......Page 184
9.9 Problems......Page 185
10.1 Convergence of frequencies to probabilities......Page 190
10.2 Sauer’s lemma......Page 192
10.3 The Vapnik–Chervonenkis theorem......Page 194
10.4 Comparison with statistical mechanics......Page 196
10.5 The Cramér–Rao inequality......Page 200
10.6 Discussion......Page 203
10.7 Problems......Page 204
11.1 The shattered coupling space......Page 207
11.2 The multifractal spectrum of the perceptron......Page 209
11.3 The multifractal organization of internal representations......Page 217
11.5 Problems......Page 221
12 Multilayer Networks......Page 223
12.1 Basic architectures......Page 224
12.2 Bounds......Page 228
12.3 The storage problem......Page 232
12.4 Generalization with a parity tree......Page 236
12.5 Generalization with a committee tree......Page 239
12.6 The fully connected committee machine......Page 242
12.7 Summary......Page 244
12.8 Problems......Page 246
13.1 The committee tree......Page 251
13.2 The parity tree......Page 257
13.3 Soft committee machine......Page 260
13.4 Back-propagation......Page 265
13.5 Bayesian on-line learning......Page 267
13.6 Discussion......Page 269
13.7 Problems......Page 270
14.1 Support vector machines......Page 273
14.2 Complex optimization......Page 277
14.3 Error-correcting codes......Page 280
14.4 Game theory......Page 284
A1.1 Gaussian integrals......Page 289
A1.2 Jensen’s inequality......Page 290
A1.3 The Delta-and Theta-functions......Page 291
A1.4 The saddle point method......Page 293
Appendix 2 The Gardner Analysis......Page 296
Appendix 3 Convergence of the Perceptron Rule......Page 303
Appendix 4 Stability of the Replica Symmetric Saddle Point......Page 305
First step......Page 307
Second step......Page 310
Third step......Page 312
Appendix 5 One-step Replica Symmetry Breaking......Page 314
Appendix 6 The Cavity Approach......Page 318
Appendix 7 The VC theorem......Page 324
Bibliography......Page 327
Index......Page 341