Coherence: In Signal Processing and Machine Learning

Preface
Contents
Acronyms
1 Introduction
	1.1 The Coherer of Hertz, Branly, and Lodge
	1.2 Interference, Coherence, and the Van Cittert-Zernike Story
	1.3 Hanbury Brown-Twiss Effect
	1.4 Tone Wobble and Coherence for Tuning
	1.5 Beampatterns and Diffraction of Electromagnetic Radiation by a Slit
	1.6 LIGO and the Detection of Einstein\'s Gravitational Waves
	1.7 Coherence and the Heisenberg Uncertainty Relations
	1.8 Coherence, Ambiguity, and the Moyal Identities
	1.9 Coherence, Correlation, and Matched Filtering
	1.10 Coherence and Matched Subspace Detectors
	1.11 What Qualifies as a Coherence?
	1.12 Why Complex?
	1.13 What is the Role of Geometry?
	1.14 Motivating Problems
	1.15 A Preview of the Book
	1.16 Chapter Notes
2 Least Squares and Related
	2.1 The Linear Model
	2.2 Over-Determined Least Squares and Related
		2.2.1 Linear Prediction
		2.2.2 Order Determination
		2.2.3 Cross-Validation
		2.2.4 Weighted Least Squares
		2.2.5 Constrained Least Squares
		2.2.6 Oblique Least Squares
		2.2.7 The BLUE (or MVUB or MVDR) Estimator
		2.2.8 Sequential Least Squares
		2.2.9 Total Least Squares
		2.2.10 Least Squares and Procrustes Problems for Channel Identification
		2.2.11 Least Squares Modal Analysis
	2.3 Under-determined Least Squares and Related
		2.3.1 Minimum-Norm Solution
		2.3.2 Sparse Solutions
		2.3.3 Maximum Entropy Solution
		2.3.4 Minimum Mean-Squared Error Solution
	2.4 Multidimensional Scaling
	2.5 The Johnson-Lindenstrauss Lemma
	2.6 Chapter Notes
	Appendices
	2.A Completing the Square in Hermitian Quadratic Forms
		2.A.1 Generalizing to Multiple Measurements and Other Cost Functions
		2.A.2 LMMSE Estimation
3 Coherence, Classical Correlations, and their Invariances
	3.1 Coherence Between a Random Variable and a Random Vector
	3.2 Coherence Between Two Random Vectors
		3.2.1 Relationship with Canonical Correlations
		3.2.2 The Circulant Case
		3.2.3 Relationship with Principal Angles
		3.2.4 Distribution of Estimated Signal-to-Noise Ratio in Adaptive Matched Filtering
	3.3 Coherence Between Two Time Series
	3.4 Multi-Channel Coherence
	3.5 Principal Component Analysis
	3.6 Two-Channel Correlation
	3.7 Multistage LMMSE Filter
	3.8 Application to Beamforming and Spectrum Analysis
		3.8.1 The Generalized Sidelobe Canceller
		3.8.2 Composite Covariance Matrix
		3.8.3 Distributions of the Conventional and Capon Beamformers
	3.9 Canonical correlation analysis
		3.9.1 Canonical Coordinates
		3.9.2 Dimension Reduction Based on Canonical and Half-Canonical Coordinates
	3.10 Partial Correlation
		3.10.1 Regressing Two Random Vectors onto One
		3.10.2 Regressing One Random Vector onto Two
	3.11 Chapter Notes
4 Coherence and Classical Tests in the Multivariate Normal Model
	4.1 How Limiting Is the Multivariate Normal Model?
	4.2 Likelihood in the MVN Model
		4.2.1 Sufficiency
		4.2.2 Likelihood
	4.3 Hypothesis Testing
	4.4 Invariance in Hypothesis Testing
	4.5 Testing for Sphericity of Random Variables
		4.5.1 Sphericity Test: Its Invariances and Null Distribution
		4.5.2 Extensions
	4.6 Testing for sphericity of random vectors
	4.7 Testing for Homogeneity of Covariance Matrices
	4.8 Testing for Independence
		4.8.1 Testing for Independence of Random Variables
		4.8.2 Testing for Independence of Random Vectors
	4.9 Cross-Validation of a Covariance Model
	4.10 Chapter Notes
5 Matched Subspace Detectors
	5.1 Signal and Noise Models
	5.2 The Detection Problem and Its Invariances
	5.3 Detectors in a First-Order Model for a Signal in a Known Subspace
		5.3.1 Scale-Invariant Matched Subspace Detector
		5.3.2 Matched Subspace Detector
	5.4 Detectors in a Second-Order Model for a Signal in a Known Subspace
		5.4.1 Scale-Invariant Matched Subspace Detector
		5.4.2 Matched Subspace Detector
	5.5 Detectors in a First-Order Model for a Signal in a Subspace Known Only by its Dimension
		5.5.1 Scale-Invariant Matched Direction Detector
		5.5.2 Matched Direction Detector
	5.6 Detectors in a Second-Order Model for a Signal in a Subspace Known Only by its Dimension
		5.6.1 Scale-Invariant Matched Direction Detector
		5.6.2 Matched Direction Detector
	5.7 Factor Analysis
	5.8 A MIMO Version of the Reed-Yu Detector
	5.9 Chapter Notes
	Appendices
	5.A Variations on Matched Subspace Detectors in a First-Order Model for a Signal in a Known Subspace
		5.A.1 Scale-Invariant, Geometrically Averaged, Matched Subspace Detector
		5.A.2 Refinement: Special Signal Sequences
		5.A.3 Rapprochement
	5.B Derivation of the Matched Subspace Detector in a Second-Order Model for a Signal in a Known Subspace
	5.C Variations on Matched Direction Detectors in a Second-Order Model for a Signal in a Subspace Known Only by its Dimension
6 Adaptive Subspace Detectors
	6.1 Introduction
	6.2 Adaptive Detection Problems
		6.2.1 Signal Models
		6.2.2 Hypothesis Tests
	6.3 Estimate and Plug (EP) Solutions for Adaptive Subspace Detection
		6.3.1 Detectors in a First-Order Model for a Signal in a Known Subspace
		6.3.2 Detectors in a Second-Order Model for a Signal in a Known Subspace
		6.3.3 Detectors in a First-Order Model for a Signal in a Subspace Known Only by its Dimension
		6.3.4 Detectors in a Second-Order Model for a Signal in a Subspace Known Only by its Dimension
	6.4 GLR Solutions for Adaptive Subspace Detection
		6.4.1 The Kelly and ACE Detector Statistics
		6.4.2 Multidimensional and Multiple Measurement GLR Extensions of the Kelly and ACE Detector Statistics
	6.5 Chapter Notes
7 Two-Channel Matched Subspace Detectors
	7.1 Signal and Noise Models for Two-Channel Problems
		7.1.1 Noise Models
		7.1.2 Known or Unknown Subspaces
	7.2 Detectors in a First-Order Model for a Signal in a Known Subspace
		7.2.1 Scale-Invariant Matched Subspace Detector for Equal and Unknown Noise Variances
		7.2.2 Matched Subspace Detector for Equal and Known Noise Variances
	7.3 Detectors in a Second-Order Model for a Signal in a Known Subspace
		7.3.1 Scale-Invariant Matched Subspace Detector for Equal and Unknown Noise Variances
		7.3.2 Scale-Invariant Matched Subspace Detector for Unequal and Unknown Noise Variances
	7.4 Detectors in a First-Order Model for a Signal in a Subspace Known Only by its Dimension
		7.4.1 Scale-Invariant Matched Direction Detector for Equal and Unknown Noise Variances
		7.4.2 Matched Direction Detector for Equal and Known Noise Variances
		7.4.3 Scale-Invariant Matched Direction Detector in Noises of Different and Unknown Variances
		7.4.4 Matched Direction Detector in Noises of Known but Different Variances
	7.5 Detectors in a Second-Order Model for a Signal in a Subspace Known Only by its Dimension
		7.5.1 Scale-Invariant Matched Direction Detector for Equal and Unknown Noise Variances
		7.5.2 Matched Direction Detector for Equal and Known Noise Variances
		7.5.3 Scale-Invariant Matched Direction Detector for Uncorrelated Noises Across Antennas (or White Noises with Different Variances)
		7.5.4 Transformation-Invariant Matched Direction Detector for Noises with Arbitrary Spatial Correlation
	7.6 Chapter Notes
8 Detection of Spatially Correlated Time Series
	8.1 Introduction
	8.2 Testing for Independence of Multiple Time Series
		8.2.1 The Detection Problem and its Invariances
		8.2.2 Test Statistic
	8.3 Approximate GLR for Multiple WSS Time Series
		8.3.1 Limiting Form of the Nonstationary GLR for WSS Time Series
		8.3.2 GLR for Multiple Circulant Time Series and an Approximate GLR for Multiple WSS Time Series
	8.4 Applications
		8.4.1 Cognitive Radio
		8.4.2 Testing for Impropriety in Time Series
	8.5 Extensions
	8.6 Detection of Cyclostationarity
		8.6.1 Problem Formulation and Its Invariances
		8.6.2 Test Statistics
		8.6.3 Interpretation of the Detectors
	8.7 Chapter Notes
9 Subspace Averaging
	9.1 The Grassmann and Stiefel Manifolds
		9.1.1 Statistics on the Grassmann and Stiefel Manifolds
	9.2 Principal Angles, Coherence, and Distances Between Subspaces
	9.3 Subspace Averages
		9.3.1 The Riemannian Mean
		9.3.2 The Extrinsic or Chordal Mean
	9.4 Order Estimation
	9.5 The Average Projection Matrix
	9.6 Application to Subspace Clustering
	9.7 Application to Array Processing
	9.8 Chapter Notes
10 Performance Bounds and Uncertainty Quantification
	10.1 Conceptual Framework
	10.2 Fisher Information and the Cramér-Rao Bound
		10.2.1 Properties of Fisher Score
		10.2.2 The Cramér-Rao Bound
		10.2.3 Geometry
	10.3 MVN Model
	10.4 Accounting for Bias
	10.5 More General Quadratic Performance Bounds
		10.5.1 Good Scores and Bad Scores
		10.5.2 Properties and Interpretations
	10.6 Information Geometry
	10.7 Chapter Notes
11 Variations on Coherence
	11.1 Coherence in Compressed Sensing
	11.2 Multiset CCA
		11.2.1 Review of Two-Channel CCA
		11.2.2 Multiset CCA (MCCA)
	11.3 Coherence in Kernel Methods
		11.3.1 Kernel Functions, Reproducing Kernel Hilbert Spaces (RKHS), and Mercer\'s Theorem
		11.3.2 Kernel CCA
		11.3.3 Coherence Criterion in KLMS
	11.4 Mutual Information as Coherence
	11.5 Coherence in Time-Frequency Modeling of a Nonstationary Time Series
	11.6 Chapter Notes
12 Epilogue
A Notation
	Sets
	Scalars, Vectors, Matrices, and Functions
	Probability, Random Variables, and Distributions
B Basic Results in Matrix Algebra
	B.1 Matrices and their Diagonalization
	B.2 Hermitian Matrices and their Eigenvalues
		B.2.1 Characterization of Eigenvalues of Hermitian Matrices
		B.2.2 Hermitian Positive Definite Matrices
	B.3 Traces
	B.4 Inverses
		B.4.1 Patterned Matrices and their Inverses
		B.4.2 Matrix Inversion Lemma or Woodbury Identity
	B.5 Determinants
		B.5.1 Some Useful Determinantal Identities and Inequalities
	B.6 Kronecker Products
	B.7 Projection Matrices
		B.7.1 Gramian, Pseudo-Inverse, and Projection
	B.8 Toeplitz, Circulant, and Hankel Matrices
	B.9 Important Matrix Optimization Problems
		B.9.1 Trace Optimization
		B.9.2 Determinant Optimization
		B.9.3 Minimize Trace or Determinant of Error Covariance in Reduced-Rank Least Squares
		B.9.4 Maximum Likelihood Estimation in a Factor Model
	B.10 Matrix Derivatives
		B.10.1 Differentiation with Respect to a Real Matrix
		B.10.2 Differentiation with Respect to a Complex Matrix
C The SVD
	C.1 The Singular Value Decomposition
	C.2 Low-Rank Matrix Approximation
	C.3 The CS Decomposition and the GSVD
		C.3.1 CS Decomposition
		C.3.2 The GSVD
D Normal Distribution Theory
	D.1 Introduction
	D.2 The Normal Random Variable
	D.3 The Multivariate Normal Random Vector
		D.3.1 Linear Transformation of a Normal Random Vector
		D.3.2 The Bivariate Normal Random Vector
		D.3.3 Analysis and Synthesis
	D.4 The Multivariate Normal Random Matrix
		D.4.1 Analysis and Synthesis
	D.5 The Spherically Invariant Bivariate Normal Experiment
		D.5.1 Coordinate Transformation: The Rayleigh and Uniform Distributions
		D.5.2 Geometry and Spherical Invariance
		D.5.3 Chi-Squared Distribution of uTu
		Beta Distribution of ρ2 = uT P1u
		D.5.5 F-Distribution of f = uT (I2−P1)u
		D.5.6 Distributions for Other Derived Random Variables
		D.5.7 Generation of Standard Normal Random Variables
	D.6 The Spherically Invariant Multivariate Normal Experiment
		D.6.1 Coordinate Transformation: The Generalized Rayleigh and Uniform Distributions
		D.6.2 Geometry and Spherical Invariance
		D.6.3 Chi-Squared Distribution of uTu
		D.6.4 Beta Distribution of ρ2p
= uT Ppu
		D.6.5 F-Distribution of fp = p
L−p
uT (IL−Pp)u
		D.6.6 Distributions for Other Derived Random Variables
	D.7 The Spherically Invariant Matrix-Valued Normal Experiment
		D.7.1 Coordinate Transformation: Bartlett\'s Factorization
		D.7.2 Geometry and Spherical Invariance
		D.7.3 Wishart Distribution of UUT
		D.7.4 The Matrix Beta Distribution
		D.7.5 The Matrix F-Distribution
		D.7.6 Special Cases
		D.7.7 Summary
	D.8 Spherical, Elliptical, and Compound Distributions
		D.8.1 Spherical Distributions
		D.8.2 Elliptical Distributions
		D.8.3 Compound Distributions
E The complex normal distribution
	E.1 The Complex MVN Distribution
	E.2 The Proper Complex MVN Distribution
	E.3 An Example from Signal Theory
	E.4 Complex Distributions
F Quadratic Forms, Cochran\'s Theorem, and Related
	F.1 Quadratic Forms and Cochran\'s Theorem
	F.2 Decomposing a Measurement into Signal and Orthogonal Subspaces
	F.3 Distribution of Squared Coherence
	F.4 Cochran\'s Theorem in the Proper Complex Case
G The Wishart distribution, the Bartlett factorization, and related
	G.1 Bartlett\'s Factorization
	G.2 Real Wishart Distribution and Related
	G.3 Complex Wishart Distribution and Related
	G.4 Distribution of Sample Mean and Sample Covariance
H Null Distribution of Coherence Statistics
	H.1 Null Distribution of the Tests for Independence
		H.1.1 Testing Independence of Random Variables
		H.1.2 Testing Independence of Random Vectors
	H.2 Testing for Block-Diagonal Matrices of Different Block Sizes
	H.3 Testing for Block-Sphericity
References
Alphabetical Index