Learning from Data. Concepts, Theory and Methods

LEARNING FROM DATA Concepts, Theory, and Methods......Page 4
CONTENTS......Page 6
PREFACE......Page 12
NOTATION......Page 18
1 Introduction......Page 20
1.1 Learning and Statistical Estimation......Page 21
1.2 Statistical Dependency and Causality......Page 26
1.3 Characterization of Variables......Page 29
1.4 Characterization of Uncertainty......Page 30
1.5 Predictive Learning versus Other Data Analytical Methodologies......Page 33
2 Problem Statement, Classical Approaches, and Adaptive Learning......Page 38
2.1 Formulation of the Learning Problem......Page 40
2.1.1 Objective of Learning......Page 43
2.1.2 Common Learning Tasks......Page 44
2.1.3 Scope of the Learning Problem Formulation......Page 48
2.2.1 Density Estimation......Page 49
2.2.2 Classification......Page 51
2.2.4 Solving Problems with Finite Data......Page 53
2.2.5 Nonparametric Methods......Page 55
2.2.6 Stochastic Approximation......Page 58
2.3.1 Philosophy, Major Concepts, and Issues......Page 59
2.3.2 A Priori Knowledge and Model Complexity......Page 62
2.3.3 Inductive Principles......Page 64
2.3.4 Alternative Learning Formulations......Page 74
2.4 Summary......Page 77
3 Regularization Framework......Page 80
3.1 Curse and Complexity of Dimensionality......Page 81
3.2 Function Approximation and Characterization of Complexity......Page 85
3.3 Penalization......Page 89
3.3.1 Parametric Penalties......Page 91
3.4 Model Selection (Complexity Control)......Page 92
3.4.1 Analytical Model Selection Criteria......Page 94
3.4.2 Model Selection via Resampling......Page 97
3.4.3 Bias–Variance Tradeoff......Page 99
3.4.4 Example of Model Selection......Page 104
3.4.5 Function Approximation versus Predictive Learning......Page 107
3.5 Summary......Page 115
4 Statistical Learning Theory......Page 118
4.1 Conditions for Consistency and Convergence of ERM......Page 120
4.2 Growth Function and VC Dimension......Page 126
4.2.1 VC Dimension for Classification and Regression Problems......Page 129
4.2.2 Examples of Calculating VC Dimension......Page 130
4.3 Bounds on the Generalization......Page 134
4.3.1 Classification......Page 135
4.3.2 Regression......Page 137
4.3.3 Generalization Bounds and Sampling Theorem......Page 139
4.4 Structural Risk Minimization......Page 141
4.4.1 Dictionary Representation......Page 143
4.4.2 Feature Selection......Page 144
4.4.4 Input Preprocessing......Page 145
4.4.5 Initial Conditions for Training Algorithm......Page 146
4.5 Comparisons of Model Selection for Regression......Page 147
4.5.1 Model Selection for Linear Estimators......Page 153
4.5.2 Model Selection for k-Nearest-Neighbor Regression......Page 156
4.5.3 Model Selection for Linear Subset Regression......Page 159
4.5.4 Discussion......Page 160
4.6 Measuring the VC Dimension......Page 162
4.7 VC Dimension, Occam’s Razor, and Popper’s Falsifiability......Page 165
4.8 Summary and Discussion......Page 168
5 Nonlinear Optimization Strategies......Page 170
5.1 Stochastic Approximation Methods......Page 173
5.1.1 Linear Parameter Estimation......Page 174
5.1.2 Backpropagation Training of MLP Networks......Page 176
5.2.1 EM Methods for Density Estimation......Page 180
5.2.2 Generalized Inverse Training of MLP Networks......Page 183
5.3.1 Neural Network Construction Algorithms......Page 188
5.3.2 Classification and Regression Trees......Page 189
5.4 Feature Selection, Optimization, and Statistical Learning Theory......Page 192
5.5 Summary......Page 194
6 Methods for Data Reduction and Dimensionality Reduction......Page 196
6.1 Vector Quantization and Clustering......Page 202
6.1.1 Optimal Source Coding in Vector Quantization......Page 203
6.1.2 Generalized Lloyd Algorithm......Page 206
6.1.3 Clustering......Page 210
6.1.4 EM Algorithm for VQ and Clustering......Page 211
6.1.5 Fuzzy Clustering......Page 214
6.2 Dimensionality Reduction: Statistical Methods......Page 220
6.2.1 Linear Principal Components......Page 221
6.2.2 Principal Curves and Surfaces......Page 224
6.2.3 Multidimensional Scaling......Page 228
6.3 Dimensionality Reduction: Neural Network Methods......Page 233
6.3.1 Discrete Principal Curves and Self-Organizing Map Algorithm......Page 234
6.3.2 Statistical Interpretation of the SOM Method......Page 237
6.3.3 Flow-Through Version of the SOM and Learning Rate Schedules......Page 241
6.3.4 SOM Applications and Modifications......Page 243
6.3.5 Self-Supervised MLP......Page 249
6.4 Methods for Multivariate Data Analysis......Page 251
6.4.1 Factor Analysis......Page 252
6.4.2 Independent Component Analysis......Page 261
6.5 Summary......Page 266
7 Methods for Regression......Page 268
7.1 Taxonomy: Dictionary versus Kernel Representation......Page 271
7.2 Linear Estimators......Page 275
7.2.1 Estimation of Linear Models and Equivalence of Representations......Page 277
7.2.2 Analytic Form of Cross-Validation......Page 281
7.2.3 Estimating Complexity of Penalized Linear Models......Page 283
7.2.4 Nonadaptive Methods......Page 288
7.3 Adaptive Dictionary Methods......Page 296
7.3.1 Additive Methods and Projection Pursuit Regression......Page 298
7.3.2 Multilayer Perceptrons and Backpropagation......Page 303
7.3.3 Multivariate Adaptive Regression Splines......Page 312
7.3.4 Orthogonal Basis Functions and Wavelet Signal Denoising......Page 317
7.4 Adaptive Kernel Methods and Local Risk Minimization......Page 328
7.4.1 Generalized Memory-Based Learning......Page 332
7.4.2 Constrained Topological Mapping......Page 333
7.5 Empirical Studies......Page 338
7.5.1 Predicting Net Asset Value (NAV) of Mutual Funds......Page 339
7.5.2 Comparison of Adaptive Methods for Regression......Page 345
7.6 Combining Predictive Models......Page 351
7.7 Summary......Page 356
8 Classification......Page 359
8.1 Statistical Learning Theory Formulation......Page 362
8.2.1 Statistical Decision Theory......Page 367
8.2.2 Fisher’s Linear Discriminant Analysis......Page 381
8.3 Methods for Classification......Page 385
8.3.1 Regression-Based Methods......Page 387
8.3.2 Tree-Based Methods......Page 397
8.3.3 Nearest-Neighbor and Prototype Methods......Page 401
8.3.4 Empirical Comparisons......Page 404
8.4 Combining Methods and Boosting......Page 409
8.4.1 Boosting as an Additive Model......Page 414
8.5 Summary......Page 420
9 Support Vector Machines......Page 423
9.1 Motivation for Margin-Based Loss......Page 427
9.2 Margin-Based Loss, Robustness, and Complexity Control......Page 434
9.3 Optimal Separating Hyperplane......Page 437
9.4 High-Dimensional Mapping and Inner Product Kernels......Page 445
9.5 Support Vector Machine for Classification......Page 449
9.6 Support Vector Implementations......Page 457
9.7 Support Vector Regression......Page 458
9.8 SVM Model Selection......Page 465
9.9 Support Vector Machines and Regularization......Page 472
9.10 Single-Class SVM and Novelty Detection......Page 479
9.11 Summary and Discussion......Page 483
10 Noninductive Inference and Alternative Learning Formulations......Page 486
10.1 Sparse High-Dimensional Data......Page 489
10.2 Transduction......Page 493
10.3 Inference Through Contradictions......Page 500
10.4 Multiple-Model Estimation......Page 505
10.5 Summary......Page 515
11 Concluding Remarks......Page 518
Appendix A: Review of Nonlinear Optimization......Page 526
Appendix B: Eigenvalues and Singular Value Decomposition......Page 533
References......Page 538
Index......Page 552