Adaptive Learning of Polynomial Networks. Genetic Programming, Backpropagation and Bayesian Methods

Contents......Page 3
Preface......Page 8
1 Introductin......Page 11
Inductive Learning......Page 13
Evolutionary Search......Page 26
STROGANOFF and its Variants......Page 27
Organization of the Book......Page 33
2 Inductive Genetic Programming......Page 35
PNN Approaches......Page 37
Tree-structured PNN......Page 39
Biological Interpretation......Page 46
Context-preserving Mutation......Page 48
Size-biasing of the Genetic Operators......Page 51
Random Tree Generation......Page 56
Schema Theorem of IGP......Page 60
Chapter Summary......Page 64
3 Tree-like PNN Representations......Page 65
Discrete Volterra Series......Page 66
Errors of Approximation  Approximation Error Bounds......Page 69
Kernel PNN Models......Page 76
Orthogonal PNN Models......Page 79
Trigonometric PNN Models......Page 81
Chapter Summary......Page 90
4 Fitness Functions & Landscapes......Page 91
Fitness Functions......Page 93
Static Fitness Functions......Page 94
Fitness Magnitude......Page 104
Statistical Correlation Measures......Page 106
Information Measures......Page 114
Quantitative Measures......Page 117
5 Search Navigation......Page 121
The Reproduction Operator......Page 122
Selection Strategies......Page 123
Replacement Strategies......Page 127
Implementing Reproduction......Page 128
Performance Examination......Page 138
Chapter Summary......Page 156
6 Backpropagation Techniques......Page 157
First-Order Backpropagation......Page 159
Batch Backpropagation......Page 167
Incremental Backpropagation......Page 168
Control of the Learning Step......Page 169
Second-Order Backpropagation......Page 173
Second-Order Error Derivatives......Page 174
Newton\'s Method......Page 179
Conjugate Gradients......Page 180
Levenberg-Marquardt Method......Page 181
First-Order Network Pruning......Page 186
Second-Order Network Pruning......Page 187
Chapter Summary......Page 189
7 Temporal Backpropagation......Page 191
Recurrent PNN as State-Space Models......Page 192
Backpropagation Through Time......Page 194
Real-Time BPTT Algorithm......Page 199
Epochwise BPTT Algorithm......Page 200
Real-Time Recurrent Learning......Page 201
Subgrouping......Page 209
Second-Order Temporal BP......Page 210
Recurrent Network Optimization......Page 216
Recurrent Network Pruning......Page 217
Chapter Summary......Page 218
8 Bayesian Inference Techniques......Page 219
Bayesian Error Function......Page 221
Deriving Hyperparameters......Page 225
Evidence Procedure for PNN Models......Page 228
Choosing a Weight Prior......Page 231
Sequential Dynamic Hessian Estimation......Page 240
Sequential Hyperparameter Estimation......Page 242
Hybrid Sampling Resampling......Page 247
Chapter Summary......Page 249
9 Statistical Model Diagnostics......Page 251
Residual Bootstrap Sampling......Page 253
Bias/Variance Dilemma......Page 254
Interval Estimation by the Delta Method......Page 258
Bootstrapping Confidence Intervals......Page 262
Prediction Intervals......Page 264
Analytical Prediction Intervals......Page 265
Empirical Learning of Prediction Bars......Page 266
Chapter Summary......Page 281
10 Time Series Modelling......Page 283
Data Preprocessing......Page 286
PNN vs. Genetically Programmed Functions......Page 289
PNN vs. Statistical Learning Networks......Page 291
PNN vs. Kernel Models......Page 295
Recurrent PNN vs. Recurrent Neural  Networks......Page 298
Chapter Summary......Page 300
11 Conclusions......Page 301
Refs......Page 305
Index......Page 322