Adaptive Filter Theory

Adaptive Filter Theory (3rd Ed.)......Page 1
Contents......Page 2
1 The Filtering Problem......Page 9
2 Adpative Filters......Page 10
3 Linear Filter Structures......Page 12
4 Approaches to Development of Linear Adaptive Filtering Algorithms......Page 17
5 Real & Complex Forms of Adaptive Filters......Page 22
6 Nonlinear Adaptive Filters......Page 23
7 Applications......Page 26
8 Some Historical Notes......Page 75
Part 1 Background Material......Page 86
1.1 z-Transform......Page 87
1.2 Linear Time-Invariant Filters......Page 89
1.3 Minimum Phase Filters......Page 94
1.5 Implementing Convolutions using DFT......Page 95
1.6 Discrete Cosine Transform......Page 101
1.7 Summary & Discussion......Page 102
Problems......Page 103
Ch2 Stationary Processes & Models......Page 104
2.1 Partial Characterization of Discrete-Time Stochastic Process......Page 105
2.2 Mean Ergodic Theorem......Page 106
2.3 Correlation Matrix......Page 108
2.4 Correlation Matrix of Sine Wave Plus Noise......Page 114
2.5 Stochastic Models......Page 116
2.6 Wold Decomposition......Page 123
2.7 Asymptotic Stationary of Autoregressive Process......Page 124
2.8 Yule-Walker Equations......Page 126
2.9 Computer Experiment: Autoregressive Process of Order 2......Page 128
2.10 Selecting the Model Order......Page 136
2.11 Complex Gaussian Processes......Page 138
2.12 Summary & Discussion......Page 140
Problems......Page 141
3.1 Power Spectral Density......Page 144
3.2 Properties of Power Spectral Density......Page 146
3.3 Transmission of Stationary Process through Linear Filter......Page 148
3.4 Cramer Spectral Representation for Stationary Process......Page 152
3.5 Power Spectrum Estimation......Page 154
3.6 Other Statistical Characteristics of Stationary Process......Page 157
3.7 Polyspectra......Page 158
3.8 Spectral-Correlation Density......Page 162
3.9 Summary & Discussion......Page 165
Problems......Page 166
4.1 The Eigenvalue Problem......Page 168
4.2 Properties of Eigenvalues & Eigenvectors......Page 170
4.3 Low-Rank Modeling......Page 184
4.4 Eigenfilters......Page 189
4.5 Eigenvalue Computations......Page 192
4.6 Summary & Discussion......Page 195
Problems......Page 196
Part 2 Linear Optimum Filtering......Page 201
5.1 Linear Optimum Filtering: Problem Statement......Page 202
5.2 Principle of Orthogonality......Page 205
5.3 Minimum Mean-Squared Error......Page 209
5.4 Wiener-Hoff Equations......Page 211
5.5 Error-Performance Surface......Page 214
5.6 Numerical Example......Page 218
5.7 Channel Equalization......Page 225
5.8 Linearly Constrained Minimum Variance Filter......Page 228
5.9 Generalized Sidelobe Cancelers......Page 235
5.10 Summary & Discussion......Page 243
Problems......Page 244
Ch6 Linear Prediction......Page 249
6.1 Forward Linear Prediction......Page 250
6.2 Backward Linear Prediction......Page 256
6.3 Levinson-Durbin Algorithm......Page 262
6.4 Properties of Prediction-Error Filters......Page 270
6.5 Schur-Cohn Test......Page 279
6.6 Autoregressive Modeling of Stationary Stochastic Process......Page 281
6.7 Cholesky Factorization......Page 284
6.8 Lattice Predictors......Page 288
6.9 Joint-Process Estimation......Page 294
6.10 Block Estimation......Page 298
6.11 Summary & Discussion......Page 301
Problems......Page 303
Ch7 Kalman Filters......Page 310
7.1 Recursive Minimum Mean-Square Estimation for Scalar Random Variables......Page 311
7.2 Statement of Kalman Filtering Problem......Page 314
7.3 The Innovations Process......Page 315
7.4 Estimation of the State using the Innovations Process......Page 318
7.5 Filtering......Page 325
7.7 Summary of Kalman Filter......Page 328
7.8 Variants of Kalman Filter......Page 330
7.9 Extended Kalman Filter......Page 336
7.10 Summary & Discussion......Page 341
Problems......Page 342
Part 3 Linear Adaptive Filtering......Page 346
8.1 Some Preliminaries......Page 347
8.2 Steepest-Descent Algorithm......Page 349
8.3 Stability of Steepest-Descent Algorithm......Page 351
8.4 Example......Page 358
Problems......Page 370
9.1 Overview of Structure & Operation of Least-Mean-Square Algorithm......Page 373
9.2 Least-Mean-Square Adaptation Algorithm......Page 375
9.3 Examples......Page 380
9.4 Stability & Performance Analysis of LMS Algorithm......Page 398
9.5 Summary of LMS Algorithm......Page 413
9.6 Computer Experiment on Adaptive Prediction......Page 414
9.7 Computer Experiment on Adaptive Equalization......Page 420
9.8 Computer Experiment on Minimum-Variance Distortionless Response Beamformer......Page 429
9.9 Directionality of Convergence of LMS Algorithm for Non-White Inputs......Page 433
9.10 Robustness of LMS Algorithm......Page 435
9.11 Normalized LMS Algorithm......Page 440
9.12 Summary & Discussion......Page 446
Problems......Page 447
Ch10 Frequency-Domain Adaptive Filters......Page 453
10.1 Block Adaptive Filters......Page 454
10.2 Fast LMS Algorithm......Page 459
10.3 Unconstrained Frequency-Domain Adaptive Filtering......Page 465
10.4 Self-Orthogonalizing Adaptive Filters......Page 466
10.5 Computer Experiment on Adaptive Equalization......Page 477
10.6 Classification of Adaptive Filtering Algorithms......Page 485
10.7 Summary & Discussion......Page 486
Problems......Page 487
11.1 Statement of Linear Least-Square Estimation Problem......Page 491
11.2 Data Windowing......Page 494
11.3 Principle of Orthogonality (Revisited)......Page 495
11.4 Minimum Sum of Error Squares......Page 499
11.5 Normal Equations & Linear Least-Squares Filters......Page 500
11.6 Time-Averaged Correlation Matrix......Page 503
11.7 Reformulation of Normal Equations in Terms of Data Matrices......Page 505
11.8 Properties of Least-Squares Estimates......Page 510
11.9 Parametric Spectrum Estimation......Page 514
11.10 Singular Value Decomposition......Page 524
11.11 Pseudoinverse......Page 532
11.12 Interpretation of Singular Values & Singular Vectors......Page 533
11.13 Minimum Norm Solution to Linear Least-Square Problem......Page 534
11.14 Normalized LMS Algorithm Viewed as Minimum-Norm Solution to Underdetermined Least-Squares Estimation Problem......Page 538
11.15 Summary & Discussion......Page 540
Problems......Page 541
Ch12 Rotations and Reflections......Page 544
12.1 Plane Rotations......Page 545
12.2 Two-Sided Jacobi Algorithm......Page 546
12.3 Cyclic Jacobi Algorithm......Page 552
12.4 Householder Transformation......Page 556
12.5 The QR Algorithm......Page 559
12.6 Summary & Discussion......Page 566
Problems......Page 568
Ch13 Recursive Least-Squares Algorithm......Page 570
13.1 Some Preliminaries......Page 571
13.2 Matrix Inversion Lemma......Page 573
13.3 Exponentially Weighted Recursive Least-Squares Algorithm......Page 574
13.4 Update Recursion for Sum of Weighted Error Squares......Page 579
13.5 Example: Single-Weight Adaptive Noise Canceler......Page 580
13.6 Convergence Analysis of RLS Algorithm......Page 581
13.7 Computer Experiment on Adaptive Equalization......Page 588
13.8 State-Space Formulation of RLS Problem......Page 591
Problems......Page 595
14.1 Square-Root Kalman Filters......Page 597
14.2 Building Square-Root Adaptive Filtering Algorithms on their Kalman Filter Counterparts......Page 605
14.3 QR-RLS Algorithm......Page 606
14.4 Extended QR-RLS Algorithm......Page 622
14.5 Adaptive Beamforming......Page 625
14.6 Inverse QR-RLS Algorithm......Page 632
14.7 Summary & Discussion......Page 635
Problems......Page 636
Ch15 Order-Recursive Adaptive Filters......Page 638
15.1 Adaptive Forward Linear Prediction......Page 639
15.2 Adaptive Backward Linear Prediction......Page 642
15.3 Conversion Factor......Page 644
15.4 Least-Squares Lattice Predictor......Page 648
15.5 Angle-Normalized Estimation Errors......Page 661
15.6 First-Order State-Space Models for Lattice Filtering......Page 663
15.7 QR-Decomposition-Based Least-Square Lattice Filters......Page 668
15.8 Fundamental Properties of QRD-LSL Filter......Page 675
15.9 Computer Experiment on Adaptive Equalization......Page 680
15.10 Extended QRD-LSL Algorithm......Page 685
15.11 Recursive Least-Squares Lattice Filters using A Posteriori Estimation Errors......Page 687
15.12 Recursive LSL Filters using A Priori Estimation Errors with Error Feedback......Page 691
15.13 Computation of Least-Squares Weight Vector......Page 694
15.14 Computer Experiment on Adaptive Prediction......Page 699
15.15 Other Variants of Least-Squares Lattice Filters......Page 701
15.16 Summary & Discussion......Page 702
Problems......Page 704
Ch16 Tracking of Time-Varying Systems......Page 709
16.1 Markov Model for System Identification......Page 710
16.2 Degree of Nonstationarity......Page 713
16.3 Criteria for Tracking Assessment......Page 714
16.4 Tracking Performance of LMS Algorithm......Page 716
16.5 Tracking Performance of RLS Algorithm......Page 719
16.6 Comparison of Tracking Performance of LMS & RLS Algorithms......Page 724
16.7 Adaptive Recovery of Chirped Sinusoidal in Noise......Page 727
16.8 How to Improve Tracking Behavior of RLS Algorithm......Page 734
16.9 Computer Experiment on System Identification......Page 737
16.10 Automatic Tuning of Adaptation Constants......Page 739
16.11 Summary & Discussion......Page 744
Problems......Page 745
Ch17 Fine-Precision Effects......Page 746
17.1 Quantization Errors......Page 747
17.2 Least-Mean-Square Algorithm......Page 749
17.3 Recursive Least-Squares Algorithm......Page 759
17.4 Square-Root Adaptive Filters......Page 765
17.5 Order-Recursive Adaptive Filters......Page 768
17.6 Fast Transversal Filters......Page 771
17.7 Summary & Discussion......Page 775
Problems......Page 777
Part 4 Nonlinear Adaptive Filtering......Page 779
Ch18 Blind Deconvolution......Page 780
18.1 Theoretical & Practical Considerations......Page 781
18.2 Bussgang Algorithm for Blind Equalization of Real Baseband Channels......Page 784
18.3 Extension of Bussgang Algorithms to Complex Baseband Channels......Page 799
18.4 Special Cases of Bussgang Algorithm......Page 800
18.5 Blind Channel Identification & Equalization using Polyspectra......Page 804
18.6 Advantage & Disadvantage of HOS-Based Deconvolution Algorithms......Page 810
18.7 Channel Identification using Cyclostationary Statistics......Page 811
18.8 Subspace Decomposition for Fractionally-Spaced Blind Identification......Page 812
18.9 Summary & Discussion......Page 821
Problems......Page 822
Ch19 Back-Propagation Learning......Page 825
19.1 Models of A Neuon......Page 826
19.2 Multilayer Perception......Page 830
19.3 Complex Back-Propagation Algorithm......Page 832
19.4 Back-Propagation Algorithm for Real Parameters......Page 845
19.5 Universal Approximation Theorem......Page 846
19.6 Network Complexity......Page 848
19.7 Filtering Applications......Page 850
19.8 Summary & Discussion......Page 860
Problems......Page 862
Ch20 Radial Basis Funuction Networks......Page 863
20.1 Structure of RBF Networks......Page 864
20.2 Radial-Basis Functions......Page 866
20.3 Fixed Centers Selected at Random......Page 867
20.4 Recursive Hybrid Learning Procedure......Page 870
20.5 Stochastic Gradient Approach......Page 871
20.6 Universal Approximation Theorem (Revisited)......Page 873
20.7 Filtering Applications......Page 874
20.8 Summary & Discussion......Page 879
Problems......Page 881
A.1 Cauthy-Riemann Equations......Page 883
A.2 Cauthy's Intergral Formula......Page 885
A.3 Laurent's Series......Page 887
A.4 Singularities & Residues......Page 889
A.5 Cauthy's Residue Theorem......Page 890
A.6 Principle of Argument......Page 892
A.7 Inversion Integral for z-Transform......Page 896
A.8 Parseval's Theorem......Page 897
B.1 Basic Definitions......Page 898
B.2 Examples......Page 900
B.3 Relation between Derivative with respect to Vector & Gradient Vector......Page 902
C.1 Optimization involving Single Equality Constraint......Page 903
C.2 Optimization involving Multiple Equality Constraints......Page 905
D.1 Likelihood Function......Page 907
D.3 Properties of Maximum-Likelihood Estimators......Page 909
D.4 Conditional Mean Estimator......Page 910
E.1 Maximum-Entropy Spectrum......Page 913
E.2 Computation of Mean Spectrum......Page 918
F.1 Fast MVDR Spectrum Computation......Page 920
F.2 Comparison of MVDR & MEM Spectra......Page 922
AppG Gradient Adaptive Lattice Algorithm......Page 923
AppH Solution of the Difference Equation (9.75)......Page 927
I.1 Iterative Solution for Weight-Error Vector......Page 929
I.2 Series Expansion of Weight-Error Correlation Matrix......Page 930
J.1 Definition......Page 932
J.2 Chi-Square Distribution as Special Case......Page 933
J.4 Expectation of Inverse Correlation Matrix......Page 935
Text Conventions......Page 936
Abbreviations......Page 940
Principal Symbols......Page 941
Bibliography......Page 949
Index......Page 986