Neural Networks: A Systematic Introduction

NEURAL NETWORKS: A SYSTEMATIC INTRODUCTION......Page 1
Web link......Page 0
Title Page......Page 2
Copyright Page......Page 3
Foreword......Page 6
Preface......Page 8
1.1.1 Natural and artificial neural networks......Page 13
1.1.2 Models of computation......Page 15
1.2 Networks of neurons......Page 19
1.2.1 Structure of the neurons......Page 20
1.2.2 Transmission of information......Page 21
1.2.3 Information processing at the neurons and synapses......Page 28
1.2.4 Storage of information – learning......Page 30
1.2.5 The neuron – a self-organizing system......Page 32
1.3.1 Networks of primitive functions......Page 33
1.3.2 Approximation of functions......Page 34
1.4 Historical and bibliographical remarks......Page 36
Exercises......Page 37
2.1.1 Feed-forward and recurrent networks......Page 39
2.1.2 The computing units......Page 41
2.2.1 Conjunction, disjunction, negation......Page 43
2.2.2 Geometric interpretation......Page 45
2.2.3 Constructive synthesis......Page 46
2.3.1 Weighted and unweighted networks......Page 48
2.3.2 Absolute and relative inhibition......Page 49
2.3.3 Binary signals and pulse coding......Page 50
2.4.1 Stored state networks......Page 52
2.4.2 Finite automata......Page 53
2.4.3 Finite automata and recurrent networks......Page 54
2.4.4 A first classification of neural networks......Page 56
2.5.1 General expression......Page 57
2.5.2 The Hadamard–Walsh transform......Page 59
2.5.3 Applications of threshold logic......Page 60
2.6 Historical and bibliographical remarks......Page 62
Exercises......Page 63
3.1.1 Perceptrons as weighted threshold elements......Page 65
3.1.2 Computational limits of the perceptron model......Page 67
3.2.1 Geometric interpretation......Page 70
3.2.2 The XOR problem......Page 72
3.3.1 Linear separability......Page 73
3.3.2 Duality of input space and weight space......Page 74
3.3.4 General decision curves......Page 75
3.4.1 Edge detection with perceptrons......Page 76
3.4.2 The structure of the retina......Page 78
3.4.3 Pyramidal networks and the neocognitron......Page 80
3.4.4 The silicon retina......Page 84
3.5 Historical and bibliographical remarks......Page 85
Exercises......Page 86
4.1 Learning algorithms for neural networks......Page 87
4.1.1 Classes of learning algorithms......Page 88
4.1.2 Vector notation......Page 89
4.1.3 Absolute linear separability......Page 90
4.1.4 The error surface and the search method......Page 91
4.2 Algorithmic learning......Page 94
4.2.1 Geometric visualization......Page 95
4.2.2 Convergence of the algorithm......Page 97
4.2.3 Accelerating convergence......Page 99
4.2.4 The pocket algorithm......Page 100
4.2.5 Complexity of perceptron learning......Page 102
4.3.1 Inner points of polytopes......Page 103
4.3.2 Linear separability as linear optimization......Page 105
4.3.3 Karmarkar’s algorithm......Page 106
4.4 Historical and bibliographical remarks......Page 108
Exercises......Page 109
5.1.1 Generalization of the perceptron problem......Page 111
5.1.2 Unsupervised learning through competition......Page 113
5.2.1 The one-dimensional case – energy function......Page 115
5.2.2 Multidimensional case – the classical methods......Page 118
5.2.3 Unsupervised learning as minimization problem......Page 120
5.2.4 Stability of the solutions......Page 123
5.3.1 Unsupervised reinforcement learning......Page 124
5.3.2 Convergence of the learning algorithm......Page 127
5.3.3 Multiple principal components......Page 129
5.4.2 Image compression......Page 130
Exercises......Page 132
6.1.1 Network architecture......Page 135
6.1.2 The XOR problem revisited......Page 136
6.1.3 Geometric visualization......Page 138
6.2.1 Weight space regions for the XOR problem......Page 141
6.2.2 Bipolar vectors......Page 143
6.2.3 Projection of the solution regions......Page 144
6.2.4 Geometric interpretation......Page 148
6.3.1 Regions in weight space for the XOR problem......Page 150
6.3.2 Number of regions in general......Page 151
6.3.3 Consequences......Page 154
6.3.4 The Vapnik–Chervonenkis dimension......Page 155
6.3.5 The problem of local minima......Page 157
6.4 Historical and bibliographical remarks......Page 159
Exercises......Page 160
7.1.1 Differentiable activation functions......Page 161
7.1.2 Regions in input space......Page 163
7.1.3 Local minima of the error function......Page 164
7.2.1 The learning problem......Page 165
7.2.2 Derivatives of network functions......Page 167
7.2.3 Steps of the backpropagation algorithm......Page 171
7.2.4 Learning with backpropagation......Page 173
7.3 The case of layered networks......Page 174
7.3.1 Extended network......Page 175
7.3.2 Steps of the algorithm......Page 176
7.3.3 Backpropagation in matrix form......Page 180
7.3.4 The locality of backpropagation......Page 182
7.4 Recurrent networks......Page 183
7.4.1 Backpropagation through time......Page 184
7.4.2 Hidden Markov Models......Page 187
7.4.3 Variational problems......Page 191
7.5 Historical and bibliographical remarks......Page 192
Exercises......Page 194
8.1 Introduction – classical backpropagation......Page 195
8.1.1 Backpropagation with momentum......Page 196
8.1.2 The fractal geometry of backpropagation......Page 203
8.2.1 Initial weight selection......Page 209
8.2.3 Reducing the number of floating-point operations......Page 212
8.2.4 Data decorrelation......Page 214
8.3 Adaptive step algorithms......Page 217
8.3.1 Silva and Almeida’s algorithm......Page 218
8.3.3 Rprop......Page 220
8.3.4 The Dynamic Adaption algorithm......Page 222
8.4 Second-order algorithms......Page 223
8.4.1 Quickprop......Page 224
8.4.2 QRprop......Page 225
8.4.3 Second-order backpropagation......Page 227
8.5.1 Weight and node perturbation......Page 234
8.5.2 Symmetric and asymmetric relaxation......Page 235
8.6 Historical and bibliographical remarks......Page 236
Exercises......Page 238
9.1.1 The problem of good generalization......Page 239
9.1.2 Linear regression......Page 241
9.1.3 Nonlinear units......Page 243
9.1.4 Computing the prediction error......Page 245
9.1.5 The jackknife and cross-validation......Page 248
9.1.6 Committees of networks......Page 249
9.2.1 Visualization of the solution regions......Page 252
9.2.2 Linear equations and the pseudoinverse......Page 254
9.2.3 The hidden layer......Page 255
9.2.4 Computation of the pseudoinverse......Page 256
9.3 Classification networks......Page 257
9.3.1 An application: NETtalk......Page 258
9.3.2 The Bayes property of classifier networks......Page 259
9.3.3 Connectionist speech recognition......Page 261
9.3.4 Autoregressive models for time series analysis......Page 270
9.4 Historical and bibliographical remarks......Page 271
Exercises......Page 272
10.1.2 Hilbert’s problem and computability......Page 275
10.1.3 Kolmogorov’s theorem......Page 277
10.2.1 The one-dimensional case......Page 279
10.2.2 The multidimensional case......Page 282
10.3 Complexity of learning problems......Page 283
10.3.1 Complexity classes......Page 284
10.3.2 NP-complete learning problems......Page 287
10.3.3 Complexity of learning with AND-OR networks......Page 289
10.3.4 Simplifications of the network architecture......Page 292
10.3.5 Learning with hints......Page 293
Exercises......Page 296
11.1.1 Imprecise data and imprecise rules......Page 299
11.1.2 The fuzzy set concept......Page 300
11.1.3 Geometric representation of fuzzy sets......Page 302
11.1.4 Fuzzy set theory, logic operators, and geometry......Page 306
11.1.5 Families of fuzzy operators......Page 307
11.2.1 Inferences from imprecise data......Page 311
11.2.2 Fuzzy numbers and inverse operation......Page 312
11.3.1 Fuzzy controllers......Page 314
11.3.2 Fuzzy networks......Page 315
11.3.3 Function approximation with fuzzy methods......Page 317
11.3.4 The eye as a fuzzy system – color vision......Page 318
11.4 Historical and bibliographical remarks......Page 319
Exercises......Page 320
12.1.1 Recurrent networks and types of associative memories......Page 321
12.1.2 Structure of an associative memory......Page 322
12.1.3 The eigenvector automaton......Page 324
12.2 Associative learning......Page 325
12.2.1 Hebbian learning – the correlation matrix......Page 326
12.2.2 Geometric interpretation of Hebbian learning......Page 329
12.2.3 Networks as dynamical systems – some experiments......Page 330
12.2.4 Another visualization......Page 334
12.3 The capacity problem......Page 335
12.4 The pseudoinverse......Page 336
12.4.1 Definition and properties of the pseudoinverse......Page 337
12.4.2 Orthogonal projections......Page 339
12.4.3 Holographic memories......Page 342
12.4.4 Translation invariant pattern recognition......Page 343
12.5 Historical and bibliographical remarks......Page 345
Exercises......Page 346
13.1.1 Recursive networks with stochastic dynamics......Page 347
13.1.2 The bidirectional associative memory......Page 348
13.1.3 The energy function......Page 350
13.2.1 Asynchronous networks......Page 351
13.2.2 Examples of the model......Page 353
13.2.3 Isomorphism between the Hopfield and Ising models......Page 358
13.3.1 Dynamics of Hopfield networks......Page 359
13.3.2 Convergence proof......Page 360
13.3.3 Hebbian learning......Page 364
13.4.1 Perceptron learning in Hopfield networks......Page 366
13.4.2 Complexity of learning in Hopfield models......Page 368
13.5.2 The multiflop problem......Page 369
13.5.3 The eight rooks problem......Page 370
13.5.4 The eight queens problem......Page 372
13.5.5 The traveling salesman......Page 373
13.5.6 The limits of Hopfield networks......Page 375
13.6.1 Electrical implementation......Page 377
13.6.2 Optical implementation......Page 378
13.7 Historical and bibliographical remarks......Page 380
Exercises......Page 381
14.1 Variations of the Hopfield model......Page 383
14.1.1 The continuous model......Page 384
14.2 Stochastic systems......Page 385
14.2.1 Simulated annealing......Page 386
14.2.2 Stochastic neural networks......Page 387
14.2.3 Markov chains......Page 388
14.2.4 The Boltzmann distribution......Page 391
14.2.5 Physical meaning of the Boltzmann distribution......Page 394
14.3.1 Boltzmann learning......Page 395
14.3.2 Combinatorial optimization......Page 397
14.4 Historical and bibliographical remarks......Page 398
Exercises......Page 399
15.1.1 Charting input space......Page 401
15.1.2 Topology preserving maps in the brain......Page 402
15.2.1 Learning algorithm......Page 405
15.2.2 Mapping high-dimensional spaces......Page 409
15.3.1 Potential function – the one-dimensional case......Page 411
15.3.2 The two-dimensional case......Page 413
15.3.3 Effect of a unit’s neighborhood......Page 414
15.3.4 Metastable states......Page 415
15.3.5 What dimension for Kohonen networks?......Page 417
15.4.1 Approximation of functions......Page 418
15.4.2 Inverse kinematics......Page 419
15.5 Historical and bibliographical remarks......Page 421
Exercises......Page 422
16.1 Constructive algorithms for modular networks......Page 423
16.1.1 Cascade correlation......Page 424
16.1.2 Optimal modules and mixtures of experts......Page 425
16.2.1 The ART architectures......Page 426
16.2.2 Maximum entropy......Page 429
16.2.3 Counterpropagation networks......Page 430
16.2.4 Spline networks......Page 432
16.2.5 Radial basis functions......Page 434
Exercises......Page 436
17.1.1 Optimization problems......Page 439
17.1.2 Methods of stochastic optimization......Page 441
17.1.3 Genetic coding......Page 443
17.2 Properties of genetic algorithms......Page 445
17.2.1 Convergence analysis......Page 446
17.2.2 Deceptive problems......Page 449
17.2.3 Genetic drift......Page 451
17.2.4 Gradient methods versus genetic algorithms......Page 452
17.3 Neural networks and genetic algorithms......Page 453
17.3.1 The problem of symmetries......Page 454
17.3.2 A numerical experiment......Page 455
17.3.3 Other applications of GAs......Page 457
17.4 Historical and bibliographical remarks......Page 458
Exercises......Page 460
18.1 Taxonomy of neural hardware......Page 461
18.1.1 Performance requirements......Page 462
18.1.2 Types of neurocomputers......Page 463
18.2.1 Coding......Page 465
18.2.2 VLSI transistor circuits......Page 466
18.2.3 Transistors with stored charge......Page 468
18.2.4 CCD components......Page 469
18.3 Digital networks......Page 471
18.3.2 Vector and signal processors......Page 472
18.3.3 Systolic arrays......Page 473
18.3.4 One-dimensional structures......Page 476
18.4.1 VLSI microprocessors for neural networks......Page 478
18.4.2 Optical computers......Page 481
18.4.3 Pulse coded networks......Page 484
18.5 Historical and bibliographical remarks......Page 486
Exercises......Page 487
References......Page 489
Back Cover......Page 512