Natural image statistics: A probabilistic approach to early computational vision

Contents......Page 9
1.1 What this Book Is All About......Page 20
1.2 What Is Vision?......Page 21
1.3 The Magic of Your Visual System......Page 22
1.4.1 Ecological Adaptation Provides Prior Information......Page 26
1.4.2 Generative Models and Latent Quantities......Page 27
1.4.4 Bayesian Inference and Priors......Page 28
1.5.1 The Image Space......Page 29
1.5.2 Definition of Natural Images......Page 30
1.6.1 Information Theory and Image Coding......Page 32
1.6.2 Redundancy Reduction and Neural Coding......Page 33
1.7.2 Normative vs. Descriptive Modeling of Visual System......Page 34
1.7.3 Toward Predictive Theoretical Neuroscience......Page 35
1.8.1 Image Representations and Features......Page 36
1.8.2 Statistics of Features......Page 37
1.8.3 From Features to Statistical Models......Page 38
1.9 The Statistical–Ecological Approach Recapitulated......Page 39
1.10 References......Page 40
Part I: Background......Page 41
2.1 Linear Filtering......Page 42
2.2 Frequency-Based Representation......Page 46
2.3 Representation Using Linear Basis......Page 55
2.4 Space-Frequency Analysis......Page 58
2.6 Exercises......Page 65
3.1 Neurons and Firing Rates......Page 67
3.2 From the Eye to the Cortex......Page 69
3.3 Linear Models of Visual Neurons......Page 70
3.4 Non-linear Models of Visual Neurons......Page 75
3.5 Interactions between Visual Neurons......Page 78
3.7 Processing after the Primary Visual Cortex......Page 80
3.9 Exercises......Page 81
4.1 Natural Images Patches as Random Vectors......Page 83
4.2 Multivariate Probability Distributions......Page 84
4.3 Marginal and Joint Probabilities......Page 86
4.4 Conditional Probabilities......Page 89
4.5 Independence......Page 91
4.6 Expectation and Covariance......Page 93
4.7 Bayesian Inference......Page 97
4.8 Parameter Estimation and Likelihood......Page 102
4.10 Exercises......Page 105
Part II: Statistics of Linear Features......Page 107
5.1 DC Component or Mean Grey-Scale Value......Page 108
5.2 Principal Component Analysis......Page 109
5.3 PCA as a Preprocessing Tool......Page 118
5.5 Gaussianity as the Basis for PCA......Page 124
5.6 Power Spectrum of Natural Images......Page 126
5.7 Anisotropy in Natural Images......Page 130
5.8 Mathematics of Principal Component Analysis*......Page 131
5.9 Decorrelation Models of Retina and LGN *......Page 135
5.10 Concluding Remarks and References......Page 143
5.11 Exercises......Page 144
6.1 Definition of Sparseness......Page 146
6.2 Learning One Feature by Maximization of Sparseness......Page 147
6.3 Learning Many Features by Maximization of Sparseness......Page 154
6.4 Sparse Coding Features for Natural Images......Page 158
6.5 How Is Sparseness Useful?......Page 162
6.6 Concluding Remarks and References......Page 163
6.7 Exercises......Page 164
7.1 Limitations of the Sparse Coding Approach......Page 166
7.2 Definition of ICA......Page 167
7.3 Insufficiency of Second-Order Information......Page 169
7.4 The Probability Density Defined by ICA......Page 173
7.5 Maximum Likelihood Estimation in ICA......Page 174
7.6 Results on Natural Images......Page 175
7.7 Connection to Maximization of Sparseness......Page 176
7.8 Why Are Independent Components Sparse?......Page 181
7.9 General ICA as Maximization of Non-Gaussianity......Page 183
7.10 Receptive Fields vs. Feature Vectors......Page 186
7.11 Problem of Inversion of Preprocessing......Page 187
7.13 Concluding Remarks and References......Page 188
7.14 Exercises......Page 189
8.1 Basic Motivation for Information Theory......Page 191
8.2 Entropy as a Measure of Uncertainty......Page 193
8.3 Mutual Information......Page 198
8.4 Minimum Entropy Coding of Natural Images......Page 199
8.5 Information Transmission in the Nervous System......Page 202
8.6 Caveats in Application of Information Theory......Page 207
8.8 Exercises......Page 209
Part III: Nonlinear Features and Dependency of Linear Features......Page 211
9.1 Why Estimated Independent Components Are Not Independent......Page 212
9.3 Modeling Using a Variance Variable......Page 214
9.4 Normalization of Variance and Contrast Gain Control......Page 216
9.5 Physical and Neurophysiological Interpretations......Page 218
9.6 Effect of Normalization on ICA......Page 220
9.7 Concluding Remarks and References......Page 223
9.8 Exercises......Page 224
10.1 Subspace Model of Invariant Features......Page 225
10.2 Maximizing Sparseness in the Energy Model......Page 228
10.3 Model of Independent Subspace Analysis......Page 231
10.4 Dependency as Energy Correlation......Page 232
10.5 Connection to Contrast Gain Control......Page 235
10.6 ISA as a Non-linear Version of ICA......Page 236
10.7 Results on Natural Images......Page 237
10.8 Analysis of Convexity and Energy Correlations*......Page 246
10.10 Exercises......Page 248
11.1 Topography in the Cortex......Page 250
11.2 Modeling Topography by Statistical Dependence......Page 251
11.3 Definition of Topographic ICA......Page 253
11.4 Connection to Independent Subspaces and Invariant Features......Page 254
11.5 Utility of Topography......Page 255
11.6 Estimation of Topographic ICA......Page 256
11.7 Topographic ICA of Natural Images......Page 257
11.8 Learning Both Layers in a Two-Layer Model *......Page 264
11.9 Concluding Remarks and References......Page 271
12.2 Simulation of V1 by a Fixed Two-Layer Model......Page 273
12.3 Learning the Third Layer by Another ICA Model......Page 275
12.4 Methods for Analyzing Higher-Order Components......Page 276
12.5 Results on Natural Images......Page 278
12.6 Discussion of Results......Page 283
12.7 Conclusion......Page 286
13.1 Overcomplete Bases......Page 287
13.2 Non-negative Models......Page 298
13.3 Conclusion......Page 303
14.1 Feedback as Bayesian Inference......Page 304
14.2 Overcomplete Basis and End-stopping......Page 311
14.3 Predictive Coding......Page 313
14.4 Conclusion......Page 314
Part IV: Time, Color, and Stereo......Page 316
15.1 Color Image Experiments......Page 317
15.2 Stereo Image Experiments......Page 323
15.3 Further References......Page 330
15.4 Conclusion......Page 331
16.1 Natural Image Sequences and Spatiotemporal Filtering......Page 332
16.2 Temporal and Spatiotemporal Receptive Fields......Page 333
16.3 Second-Order Statistics......Page 335
16.4 Sparse Coding and ICA of Natural Image Sequences......Page 340
16.5 Temporal Coherence in Spatial Features......Page 343
16.6 Spatiotemporal Energy Correlations in Linear Features......Page 352
16.7 Unifying Model of Spatiotemporal Dependencies......Page 359
16.8 Features with Minimal Average Temporal Change......Page 361
16.9 Conclusion......Page 368
Part V: Conclusion......Page 369
17.1 Short Overview......Page 370
17.2 Open, or Frequently Asked, Questions......Page 372
17.3 Other Mathematical Models of Images......Page 376
17.4 Future Work......Page 379
Part VI: Appendix: Supplementary Mathematical Tools......Page 380
18.1 Levels of Modeling......Page 381
18.2 Gradient Method......Page 382
18.4 Hebb\'s Rule and Gradient Methods......Page 388
18.5 Optimization in Topographic ICA *......Page 393
18.6 Beyond Basic Gradient Methods *......Page 394
18.7 FastICA, a Fixed-Point Algorithm for ICA......Page 398
19.1 Vectors......Page 402
19.2 Linear Transformations......Page 403
19.3 Matrices......Page 404
19.5 Inverse......Page 405
19.6 Basis Representations......Page 406
19.7 Orthogonality......Page 407
19.8 Pseudo-Inverse *......Page 408
20.1 Linear Shift-Invariant Systems......Page 409
20.2 One-Dimensional Discrete Fourier Transform......Page 410
20.3 Two- and Three-Dimensional Discrete Fourier Transforms......Page 419
21.1 Non-normalized Statistical Models......Page 421
21.2 Estimation by Score Matching......Page 422
21.3 Example 1: Multivariate Gaussian Density......Page 424
21.4 Example 2: Estimation of Basic ICA Model......Page 426
21.6 Conclusion......Page 427
References......Page 429
C......Page 443
F......Page 444
I......Page 445
M......Page 446
P......Page 447
S......Page 448
V......Page 449
W......Page 450