Statistical signal processing in engineering

Content: List of Figures xvii List of Tables xxiii Preface xxv List of Abbreviations xxix How to Use the Book xxxi About the Companion Website xxxiii Prerequisites xxxv Why are there so many matrixes in this book? xxxvii 1 Manipulations on Matrixes 1 1.1 Matrix Properties 1 1.1.1 Elementary Operations 2 1.2 Eigen-Decomposition 6 1.3 Eigenvectors in Everyday Life 9 1.3.1 Conversations in a Noisy Restaurant 9 1.3.2 Power Control in a Cellular System 12 1.3.3 Price Equilibrium in the Economy 14 1.4 Derivative Rules 15 1.4.1 Derivative with respect to x 16 1.4.2 Derivative with respect to x 17 1.4.3 Derivative with respect to the Matrix X 18 1.5 Quadratic Forms 19 1.6 Diagonalization of a Quadratic Form 20 1.7 Rayleigh Quotient 21 1.8 Basics of Optimization 22 1.8.1 Quadratic Function with Simple Linear Constraint (M=1) 23 1.8.2 Quadratic Function with Multiple Linear Constraints 23 Appendix A: Arithmetic vs. Geometric Mean 24 2 Linear Algebraic Systems 27 2.1 Problem Definition and Vector Spaces 27 2.1.1 Vector Spaces in Tomographic Radiometric Inversion 29 2.2 Rotations 31 2.3 Projection Matrixes and Data-Filtering 33 2.3.1 Projections and Commercial FM Radio 34 2.4 Singular Value Decomposition (SVD) and Subspaces 34 2.4.1 How to Choose the Rank of Afor Gaussian Model? 35 2.5 QR and Cholesky Factorization 36 2.6 Power Method for Leading Eigenvectors 38 2.7 Least Squares Solution of Overdetermined Linear Equations 39 2.8 Efficient Implementation of the LS Solution 41 2.9 Iterative Methods 42 3 Random Variables in Brief 45 3.1 Probability Density Function (pdf), Moments, and Other Useful Properties 45 3.2 Convexity and Jensen Inequality 49 3.3 Uncorrelatedness and Statistical Independence 49 3.4 Real-Valued Gaussian Random Variables 51 3.5 Conditional pdf for Real-Valued Gaussian Random Variables 54 3.6 Conditional pdf in Additive Noise Model 56 3.7 Complex Gaussian Random Variables 56 3.7.1 Single Complex Gaussian Random Variable 56 3.7.2 Circular Complex Gaussian Random Variable 57 3.7.3 Multivariate Complex Gaussian Random Variables 58 3.8 Sum of Square of Gaussians: Chi-Square 59 3.9 Order Statistics for N rvs 60 4 Random Processes and Linear Systems 63 4.1 Moment Characterizations and Stationarity 64 4.2 Random Processes and Linear Systems 66 4.3 Complex-Valued Random Processes 68 4.4 Pole-Zero and Rational Spectra (Discrete-Time) 69 4.4.1 Stability of LTI Systems 70 4.4.2 Rational PSD 71 4.4.3 Paley   Wiener Theorem 72 4.5 Gaussian Random Process (Discrete-Time) 73 4.6 Measuring Moments in Stochastic Processes 75 Appendix A: Transforms for Continuous-Time Signals 76 Appendix B: Transforms for Discrete-Time Signals 79 5 Models and Applications 83 5.1 Linear Regression Model 84 5.2 Linear Filtering Model 86 5.2.1 Block-Wise Circular Convolution 88 5.2.2 Discrete Fourier Transform and Circular Convolution Matrixes 89 5.2.3 Identification and Deconvolution 90 5.3 MIMO systems and Interference Models 91 5.3.1 DSL System 92 5.3.2 MIMO in Wireless Communication 92 5.4 Sinusoidal Signal 97 5.5 Irregular Sampling and Interpolation 97 5.5.1 Sampling With Jitter 100 5.6 Wavefield Sensing System 101 6 Estimation Theory 105 6.1 Historical Notes 105 6.2 Non-Bayesian vs. Bayesian 106 6.3 Performance Metrics and Bounds 107 6.3.1 Bias 107 6.3.2 Mean Square Error (MSE) 108 6.3.3 Performance Bounds 109 6.4 Statistics and Sufficient Statistics 110 6.5 MVU and BLU Estimators 111 6.6 BLUE for Linear Models 112 6.7 Example: BLUE of the Mean Value of Gaussian rvs 114 7 Parameter Estimation 117 7.1 Maximum Likelihood Estimation (MLE) 117 7.2 MLE for Gaussian Model 119 7.2.1 Additive Noise Model with 119 7.2.2 Additive Noise Model with 120 7.2.3 Additive Noise Model with Multiple Observations with Known 121 7.2.3.1 Linear Model 121 7.2.3.2 Model 122 7.2.3.3 Model 123 7.2.4 Model 123 7.2.5 Additive Noise Model with Multiple Observations with Unknown 124 7.3 Other Noise Models 125 7.4 MLE and Nuisance Parameters 126 7.5 MLE for Continuous-Time Signals 128 7.5.1 Example: Amplitude Estimation 129 7.5.2 MLE for Correlated Noise 130 7.6 MLE for Circular Complex Gaussian 131 7.7 Estimation in Phase/Frequency Modulations 131 7.7.1 MLE Phase Estimation 132 7.7.2 Phase Locked Loops 133 7.8 Least Square (LS) Estimation 135 7.8.1 Weighted LS with 136 7.8.2 LS Estimation and Linear Models 137 7.8.3 Under or Over-Parameterizing? 138 7.8.4 Constrained LS Estimation 139 7.9 Robust Estimation 140 8 Cramer   Rao Bound 143 8.1 Cramer   Rao Bound and Fisher Information Matrix 143 8.1.1 CRB for Scalar Problem (P=1) 143 8.1.2 CRB and Local Curvature of Log-Likelihood 144 8.1.3 CRB for Multiple Parameters (p 1) 144 8.2 Interpretation of CRB and Remarks 146 8.2.1 Variance of Each Parameter 146 8.2.2 Compactness of the Estimates 146 8.2.3 FIM for Known Parameters 147 8.2.4 Approximation of the Inverse of FIM 148 8.2.5 Estimation Decoupled From FIM 148 8.2.6 CRB and Nuisance Parameters 149 8.2.7 CRB for Non-Gaussian rv and Gaussian Bound 149 8.3 CRB and Variable Transformations 150 8.4 FIM for Gaussian Parametric Model 151 8.4.1 FIM for with 151 8.4.2 FIM for Continuous-Time Signals in Additive White Gaussian Noise 152 8.4.3 FIM for Circular Complex Model 152 Appendix A: Proof of CRB 154 Appendix B: FIM for Gaussian Model 156 Appendix C: Some Derivatives for MLE and CRB Computations 157 9 MLE and CRB for Some Selected Cases 159 9.1 Linear Regressions 159 9.2 Frequency Estimation 162 9.3 Estimation of Complex Sinusoid 164 9.3.1 Proper, Improper, and Non-Circular Signals 165 9.4 Time of Delay Estimation 166 9.5 Estimation of Max for Uniform pdf 170 9.6 Estimation of Occurrence Probability for Binary pdf 172 9.7 How to Optimize Histograms? 173 9.8 Logistic Regression 176 10 Numerical Analysis and Montecarlo Simulations 179 10.1 System Identification and Channel Estimation 181 10.1.1 Matlab Code and Results 184 10.2 Frequency Estimation 184 10.2.1 Variable (Coarse/Fine) Sampling 187 10.2.2 Local Parabolic Regression 189 10.2.3 Matlab Code and Results 190 10.3 Time of Delay Estimation 192 10.3.1 Granularity of Sampling in ToD Estimation 193 10.3.2 Matlab Code and Results 194 10.4 Doppler-Radar System by Frequency Estimation 196 10.4.1 EM Method 197 10.4.2 Matlab Code and Results 199 11 Bayesian Estimation 201 11.1 Additive Linear Model with Gaussian Noise 203 11.1.1 Gaussian A-priori: 204 11.1.2 Non-Gaussian A-Priori 206 11.1.3 Binary Signals: MMSE vs. MAP Estimators 207 11.1.4 Example: Impulse Noise Mitigation 210 11.2 Bayesian Estimation in Gaussian Settings 212 11.2.1 MMSE Estimator 213 11.2.2 MMSE Estimator for Linear Models 213 11.3 LMMSE Estimation and Orthogonality 215 11.4 Bayesian CRB 218 11.5 Mixing Bayesian and Non-Bayesian 220 11.5.1 Linear Model with Mixed Random/Deterministic Parameters 220 11.5.2 Hybrid CRB 222 11.6 Expectation-Maximization (EM) 223 11.6.1 EM of the Sum of Signals in Gaussian Noise 224 11.6.2 EM Method for the Time of Delay Estimation of Multiple Waveforms 227 11.6.3 Remarks 228 Appendix A: Gaussian Mixture pdf 229 12 Optimal Filtering 231 12.1 Wiener Filter 231 12.2 MMSE Deconvolution (or Equalization) 233 12.3 Linear Prediction 234 12.3.1 Yule   Walker Equations 235 12.4 LS Linear Prediction 237 12.5 Linear Prediction and AR Processes 239 12.6 Levinson Recursion and Lattice Predictors 241 13 Bayesian Tracking and Kalman Filter 245 13.1 Bayesian Tracking of State in Dynamic Systems 246 13.1.1 Evolution of the A-posteriori pdf 247 13.2 Kalman Filter (KF) 249 13.2.1 KF Equations 251 13.2.2 Remarks 253 13.3 Identification of Time-Varying Filters in Wireless Communication 255 13.4 Extended Kalman Filter (EKF) for Non-Linear Dynamic Systems 257 13.5 Position Tracking by Multi-Lateration 258 13.5.1 Positioning and Noise 260 13.5.2 Example of Position Tracking 263 13.6 Non-Gaussian Pdf and Particle Filters264 14 Spectral Analysis 267 14.1 Periodogram 268 14.1.1 Bias of the Periodogram 268 14.1.2 Variance of the Periodogram 271 14.1.3 Filterbank Interpretation 273 14.1.4 Pdf of the Periodogram (White Gaussian Process) 274 14.1.5 Bias and Resolution 275 14.1.6 Variance Reduction and WOSA 278 14.1.7 Numerical Example: Bandlimited Process and (Small) Sinusoid 280 14.2 Parametric Spectral Analysis 282 14.2.1 MLE and CRB 284 14.2.2 General Model for AR, MA, ARMA Spectral Analysis 285 14.3 AR Spectral Analysis 286 14.3.1 MLE and CRB 286 14.3.2 A Good Reason to Avoid Over-Parametrization in AR 289 14.3.3 Cramer   Rao Bound of Poles in AR Spectral Analysis 291 14.3.4 Example: Frequency Estimation by AR Spectral Analysis 293 14.4 MA Spectral Analysis 296 14.5 ARMA Spectral Analysis 298 14.5.1 Cramer   Rao Bound for ARMA Spectral Analysis 300 Appendix A: Which Sample Estimate of the Autocorrelation to Use? 302 Appendix B: Eigenvectors and Eigenvalues of Correlation Matrix 303 Appendix C: Property of Monic Polynomial 306 Appendix D: Variance of Pole in AR(1) 307 15 Adaptive Filtering 309 15.1 Adaptive Interference Cancellation 311 15.2 Adaptive Equalization in Communication Systems 313 15.2.1 Wireless Communication Systems in Brief 313 15.2.2 Adaptive Equalization 315 15.3 Steepest Descent MSE Minimization 317 15.3.1 Convergence Analysis and Step-Size 318 15.3.2 An Intuitive View of Convergence Conditions 320 15.4 From Iterative to Adaptive Filters 323 15.5 LMS Algorithm and Stochastic Gradient 324 15.6 Convergence Analysis of LMS Algorithm 325 15.6.1 Convergence in the Mean 326 15.6.2 Convergence in the Mean Square 326 15.6.3 Excess MSE 329 15.7 Learning Curve of LMS 331 15.7.1 Optimization of the Step-Size 332 15.8 NLMS Updating and Non-Stationarity 333 15.9 Numerical Example: Adaptive Identification 334 15.10 RLS Algorithm 338 15.10.1 Convergence Analysis 339 15.10.2 Learning Curve of RLS 341 15.11 Exponentially-Weighted RLS 342 15.12 LMS vs. RLS 344 Appendix A: Convergence in Mean Square 344 16 Line Spectrum Analysis 347 16.1 Model Definition 349 16.1.1 Deterministic Signals 350 16.1.2 Random Signals 350 16.1.3 Properties of Structured Covariance 351 16.2 Maximum Likelihood and Cramer   Rao Bounds 352 16.2.1 Conditional ML 353 16.2.2 Cramer   Rao Bound for Conditional Model 354 16.2.3 Unconditional ML 356 16.2.4 Cramer   Rao Bound for Unconditional Model 356 16.2.5 Conditional vs. Unconditional Model & Bounds 357 16.3 High-Resolution Methods 357 16.3.1 Iterative Quadratic ML (IQML) 358 16.3.2 Prony Method 360 16.3.3 MUSIC 360 16.3.4 ESPRIT 363 16.3.5 Model Order 365 17 Equalization in Communication Engineering 367 17.1 Linear Equalization 369 17.1.1 Zero Forcing (ZF) Equalizer 370 17.1.2 Minimum Mean Square Error (MMSE) Equalizer 371 17.1.3 Finite-Length/Finite-Block Equalizer 371 17.2 Non-Linear Equalization 372 17.2.1 ZF-DFE 373 17.2.2 MMSE   DFE 374 17.2.3 Finite-Length MMSE   DFE 375 17.2.4 Asymptotic Performance for Infinite-Length Equalizers 376 17.3 MIMO Linear Equalization 377 17.3.1 ZF MIMO Equalization 377 17.3.2 MMSE MIMO Equalization 379 17.4 MIMO   DFE Equalization 379 17.4.1 Cholesky Factorization and Min/Max Phase Decomposition 379 17.4.2 MIMO   DFE 380 18 2D Signals and Physical Filters 383 18.1 2D Sinusoids 384 18.1.1 Moire Pattern 386 18.2 2D Filtering 388 18.2.1 2D Random Fields 390 18.2.2 Wiener Filtering 391 18.2.3 Image Acquisition and Restoration 392 18.3 Diffusion Filtering 394 18.3.1 Evolution vs. Time: Fourier Method 394 18.3.2 Extrapolation of the Density 395 18.3.3 Effect of Phase-Shift 396 18.4 Laplace Equation and Exponential Filtering 398 18.5 Wavefield Propagation 400 18.5.1 Propagation/Backpropagation 400 18.5.2 Wavefield Extrapolation and Focusing 402 18.5.3 Exploding Reflector Model 402 18.5.4 Wavefield Extrapolation 404 18.5.5 Wavefield Focusing (or Migration) 406 Appendix A: Properties of 2D Signals 406 Appendix B: Properties of 2D Fourier Transform 410 Appendix C: Finite Difference Method for PDE-Diffusion 412 19 Array Processing 415 19.1 Narrowband Model 415 19.1.1 Multiple DoAs and Multiple Sources 419 19.1.2 Sensor Spacing Design 420 19.1.3 Spatial Resolution and Array Aperture 421 19.2 Beamforming and Signal Estimation 422 19.2.1 Conventional Beamforming 425 19.2.2 Capon Beamforming (MVDR) 426 19.2.3 Multiple-Constraint Beamforming 429 19.2.4 Max-SNR Beamforming 431 19.3 DoA Estimation 432 19.3.1 ML Estimation and CRB 433 19.3.2 Beamforming and Root-MVDR 434 20 Multichannel Time of Delay Estimation 435 20.1 Model Definition for ToD 440 20.2 High Resolution Method for ToD (L=1) 441 20.2.1 ToD in the Fourier Transformed Domain 441 20.2.2 CRB and Resolution 444 20.3 Difference of ToD (DToD) Estimation 445 20.3.1 Correlation Method for DToD 445 20.3.2 Generalized Correlation Method 448 20.4 Numerical Performance Analysis of DToD 452 20.5 Wavefront Estimation: Non-Parametric Method (L=1) 454 20.5.1 Wavefront Estimation in Remote Sensing and Geophysics 456 20.5.2 Narrowband Waveforms and 2D Phase Unwrapping 457 20.5.3 2D Phase Unwrapping in Regular Grid Spacing 458 20.6 Parametric ToD Estimation and Wideband Beamforming 460 20.6.1 Delay and Sum Beamforming 462 20.6.2 Wideband Beamforming After Fourier Transform 464 20.7 Appendix A: Properties of the Sample Correlations 465 20.8 Appendix B: How to Delay a Discrete-Time Signal? 466 20.9 Appendix C: Wavefront Estimation for 2D Arrays 467 21 Tomography 467 21.1 X-ray Tomography 471 21.1.1 Discrete Model 471 21.1.2 Maximum Likelihood 473 21.1.3 Emission Tomography 473 21.2 Algebraic Reconstruction Tomography (ART) 475 21.3 Reconstruction From Projections: Fourier Method 475 21.3.1 Backprojection Algorithm 476 21.3.2 How Many Projections to Use? 479 21.4 Traveltime Tomography 480 21.5 Internet (Network) Tomography 483 21.5.1 Latency Tomography 484 21.5.2 Packet-Loss Tomography 484 22 Cooperative Estimation 487 22.1 Consensus and Cooperation 490 22.1.1 Vox Populi: The Wisdom of Crowds 490 22.1.2 Cooperative Estimation as Simple Information Consensus 490 22.1.3 Weighted Cooperative Estimation ( ) 493 22.1.4 Distributed MLE ( ) 495 22.2 Distributed Estimation for Arbitrary Linear Models (p>
1) 496 22.2.1 Centralized MLE 497 22.2.2 Distributed Weighted LS 498 22.2.3 Distributed MLE 500 22.2.4 Distributed Estimation for Under-Determined Systems 501 22.2.5 Stochastic Regressor Model 503 22.2.6 Cooperative Estimation in the Internet of Things (IoT) 503 22.2.7 Example: Iterative Distributed Estimation 505 22.3 Distributed Synchronization 506 22.3.1 Synchrony-States for Analog and Discrete-Time Clocks 507 22.3.2 Coupled Clocks 510 22.3.3 Internet Synchronization and the Network Time Protocol (NTP) 512 Appendix A: Basics of Undirected Graphs 515 23 Classification and Clustering 521 23.1 Historical Notes 522 23.2 Classification 523 23.2.1 Binary Detection Theory 523 23.2.2 Binary Classification of Gaussian Distributions 528 23.3 Classification of Signals in Additive Gaussian Noise 529 23.3.1 Detection of Known Signal 531 23.3.2 Classification of Multiple Signals 532 23.3.3 Generalized Likelihood Ratio Test (GLRT) 533 23.3.4 Detection of Random Signals 535 23.4 Bayesian Classification 536 23.4.1 To Classify or Not to Classify? 537 23.4.2 Bayes Risk 537 23.5 Pattern Recognition and Machine Learning 538 23.5.1 Linear Discriminant 539 23.5.2 Least Squares Classification 540 23.5.3 Support Vectors Principle 541 23.6 Clustering 543 23.6.1 K-Means Clustering 544 23.6.2 EM Clustering 545 References 549 Index 557