Matrix Algebra: Theory, Computations and Applications in Statistics (Springer Texts in Statistics)

Preface to Third Edition
Preface to Second Edition
Preface to First Edition
1 Introduction
	1.1 Vectors
	1.2 Arrays
	1.3 Matrices
	1.4 Representation of Data
	1.5 What You Compute and What You Don\'t
	1.6 R
		1.6.1 R Data Types
		1.6.2 Program Control Statements
		1.6.3 Packages
		1.6.4 User Functions and Operators
		1.6.5 Generating Artificial Data
		1.6.6 Graphics Functions
		1.6.7 Special Data in R
		1.6.8 Determining Properties of a Computer System in R
		1.6.9 Documentation: Finding R Functions and Packages
		1.6.10 Documentation: R Functions, Packages, and Other Objects
		1.6.11 The Design of the R Programming Language
		1.6.12 Why I Use R in This Book
Part I Linear Algebra
	2 Vectors and Vector Spaces
		2.1 Operations on Vectors
			2.1.1 Linear Combinations and Linear Independence
			2.1.2 Vector Spaces and Spaces of Vectors
				Generating Sets
				The Order and the Dimension of a Vector Space
				Vector Spaces with an Infinite Number of Dimensions
				Some Special Vectors: Notation
				Ordinal Relations Among Vectors
				Set Operations on Vector Spaces
				Essentially Disjoint Vector Spaces
				Subpaces
				Intersections of Vector Spaces
				Unions and Direct Sums of Vector Spaces
				Direct Sum Decomposition of a Vector Space
				Direct Products of Vector Spaces and Dimension Reduction
			2.1.3 Basis Sets for Vector Spaces
				Properties of Basis Sets of Vector Subspaces
			2.1.4 Inner Products
				Inner Products in General Real Vector Spaces
				The Inner Product in Real Vector Spaces
			2.1.5 Norms
				Convexity
				Norms Induced by Inner Products
				Lp Norms
				Basis Norms
				Equivalence of Norms
			2.1.6 Normalized Vectors
				``Inverse\'\' of a Vector
			2.1.7 Metrics and Distances
				Metrics Induced by Norms
				Convergence of Sequences of Vectors
			2.1.8 Orthogonal Vectors and Orthogonal Vector Spaces
			2.1.9 The ``One Vector\'\'
				The Mean and the Mean Vector
		2.2 Cartesian Coordinates and Geometrical Properties of Vectors
			2.2.1 Cartesian Geometry
			2.2.2 Projections
			2.2.3 Angles Between Vectors
			2.2.4 Orthogonalization Transformations: Gram–Schmidt
			2.2.5 Orthonormal Basis Sets
			2.2.6 Approximation of Vectors
				Optimality of the Fourier Coefficients
				Choice of the Best Basis Subset
			2.2.7 Flats, Affine Spaces, and Hyperplanes
			2.2.8 Cones
				Convex Cones
				Dual Cones
				Polar Cones
				Additional Properties
			2.2.9 Vector Cross Products in IR3
		2.3 Centered Vectors, and Variances and Covariances of Vectors
			2.3.1 The Mean and Centered Vectors
			2.3.2 The Standard Deviation, the Variance, and Scaled Vectors
			2.3.3 Covariances and Correlations Between Vectors
		Appendix: Vectors in R
		Exercises
	3 Basic Properties of Matrices
		3.1 Basic Definitions and Notation
			3.1.1 Multiplication of a Matrix by a Scalar
			3.1.2 Symmetric and Hermitian Matrices
			3.1.3 Diagonal Elements
			3.1.4 Diagonally Dominant Matrices
			3.1.5 Diagonal and Hollow Matrices
			3.1.6 Matrices with Other Special Patterns of Zeroes
			3.1.7 Matrix Shaping Operators
				Transpose
				Conjugate Transpose
				Properties
				Diagonals of Matrices and Diagonal Vectors: vecdiag(·) or diag(·)
				The Diagonal Matrix Constructor Function diag(·)
				Forming a Vector from the Elements of a Matrix: vec(·) and vech(·)
			3.1.8 Partitioned Matrices and Submatrices
				Notation
				Block Diagonal Matrices
				The diag(·) Matrix Function and the Direct Sum
				Transposes of Partitioned Matrices
			3.1.9 Matrix Addition
				The Transpose of the Sum of Matrices
				Rank Ordering Matrices
				Vector Spaces of Matrices
			3.1.10 The Trace of a Square Matrix
				The Trace: tr(·)
				The Trace of the Transpose of Square Matrices
				The Trace of Scalar Products of Square Matrices
				The Trace of Partitioned Square Matrices
				The Trace of the Sum of Square Matrices
		3.2 The Determinant
			3.2.1 Definition and Simple Properties
				Notation
			3.2.2 Determinants of Various Types of Square Matrices
				The Determinant of the Transpose of Square Matrices
				The Determinant of Scalar Products of Square Matrices
				The Determinant of a Triangular Matrix, Upper or Lower
				The Determinant of Square Block Triangular Matrices
				The Determinant of the Sum of Square Matrices
			3.2.3 Minors, Cofactors, and Adjugate Matrices
				An Expansion of the Determinant
				Definitions of Terms: Minors and Cofactors
				Adjugate Matrices
				Cofactors and Orthogonal Vectors
				A Diagonal Expansion of the Determinant
			3.2.4 A Geometrical Perspective of the Determinant
				Computing the Determinant
		3.3 Multiplication of Matrices and Multiplication of  Vectors and Matrices
			3.3.1 Matrix Multiplication (Cayley)
				Factorization of Matrices
				Powers of Square Matrices
				Idempotent and Nilpotent Matrices
				Matrix Polynomials
			3.3.2 Cayley Multiplication of Matrices with Special Patterns
				Multiplication of Matrices and Vectors
				The Matrix/Vector Product as a Linear Combination
				The Matrix as a Mapping on Vector Spaces
				Multiplication of Partitioned Matrices
			3.3.3 Elementary Operations on Matrices
				Interchange of Rows or Columns: Permutation Matrices
				The Vec-Permutation Matrix
				Scalar Row or Column Multiplication
				Axpy Row or Column Transformations
				Gaussian Elimination: Gaussian Transformation Matrix
				Elementary Operator Matrices: Summary of Notation and Properties
				Determinants of Elementary Operator Matrices
			3.3.4 The Trace of a Cayley Product that Is Square
			3.3.5 The Determinant of a Cayley Product of Square Matrices
				The Adjugate in Matrix Multiplication
			3.3.6 Outer Products of Vectors
			3.3.7 Bilinear and Quadratic Forms: Definiteness
				Nonnegative Definite and Positive Definite Matrices
				Ordinal Relations among Symmetric Matrices
				The Trace of Inner and Outer Products
			3.3.8 Anisometric Spaces
				Conjugacy
			3.3.9 The Hadamard Product
			3.3.10 The Kronecker Product
			3.3.11 The Inner Product of Matrices
				Orthonormal Matrices
				Orthonormal Basis: Fourier Expansion
		3.4 Matrix Rank and the Inverse of a Matrix
			3.4.1 Row Rank and Column Rank
			3.4.2 Full Rank Matrices
			3.4.3 Rank of Elementary Operator Matrices and Matrix Products Involving Them
			3.4.4 The Rank of Partitioned Matrices, Products of Matrices, and Sums of Matrices
				Rank of Partitioned Matrices and Submatrices
				An Upper Bound on the Rank of Products of Matrices
				An Upper and a Lower Bound on the Rank of Sums of Matrices
			3.4.5 Full Rank Partitioning
			3.4.6 Full Rank Matrices and Matrix Inverses
				Solutions of Linear Equations
				Consistent Systems
				Multiple Consistent Systems
			3.4.7 Matrix Inverses
				The Inverse of a Square Nonsingular Matrix
				The Inverse and the Solution to a Linear System
				Inverses and Transposes
				Nonsquare Full Rank Matrices: Right and Left Inverses
			3.4.8 Full Rank Factorization
			3.4.9 Multiplication by Full Rank Matrices
				Products with a Nonsingular Matrix
				Products with a General Full Rank Matrix
				Preservation of Positive Definiteness
				The General Linear Group
			3.4.10 Nonfull Rank and Equivalent Matrices
				Equivalent Canonical Forms
				A Factorization Based on an Equivalent Canonical Form
				Equivalent Forms of Symmetric Matrices
			3.4.11 Gramian Matrices: Products of the Form ATA
				General Properties of Gramian Matrices
				Rank of ATA
				Zero Matrices and Equations Involving Gramians
			3.4.12 A Lower Bound on the Rank of a Matrix Product
			3.4.13 Determinants of Inverses
			3.4.14 Inverses of Products and Sums of Nonsingular Matrices
				Inverses of Cayley Products of Matrices
				Inverses of Kronecker Products of Matrices
				Inverses of Sums of Matrices and Their Inverses
				An Expansion of a Matrix Inverse
			3.4.15 Inverses of Matrices with Special Forms
			3.4.16 Determining the Rank of a Matrix
		3.5 The Schur Complement in Partitioned Square Matrices
			3.5.1 Inverses of Partitioned Matrices
			3.5.2 Determinants of Partitioned Matrices
		3.6 Linear Systems of Equations
			3.6.1 Solutions of Linear Systems
				Underdetermined Systems
				Overdetermined Systems
				Solutions in Consistent Systems: Generalized Inverses
			3.6.2 Null Space: The Orthogonal Complement
			3.6.3 Orthonormal Completion
		3.7 Generalized Inverses
			3.7.1 Immediate Properties of Generalized Inverses
				Rank and the Reflexive Generalized Inverse
				Properties of Generalized Inverses Useful in Analysis of Linear Models
			3.7.2 The Moore–Penrose Inverse
				Definitions and Terminology
				Existence
				Uniqueness
				Other Properties
			3.7.3 Generalized Inverses of Products and Sums of Matrices
			3.7.4 Generalized Inverses of Partitioned Matrices
		3.8 Orthogonality
			3.8.1 Orthogonal Matrices: Definition and Simple Properties
			3.8.2 Unitary Matrices
			3.8.3 Orthogonal and Orthonormal Columns
			3.8.4 The Orthogonal Group
			3.8.5 Conjugacy
		3.9 Eigenanalysis: Canonical Factorizations
			3.9.1 Eigenvalues and Eigenvectors Are Remarkable
			3.9.2 Basic Properties of Eigenvalues and Eigenvectors
				Eigenvalues of Elementary Operator Matrices
				Left Eigenvectors
			3.9.3 The Characteristic Polynomial
				How Many Eigenvalues Does a Matrix Have?
				Properties of the Characteristic Polynomial
				Additional Properties of Eigenvalues and Eigenvectors
				Eigenvalues and the Trace and the Determinant
			3.9.4 The Spectrum
				Notation
				The Spectral Radius
				Linear Independence of Eigenvectors Associated with  Distinct Eigenvalues
				The Eigenspace and Geometric Multiplicity
				Algebraic Multiplicity
				Gershgorin Disks
			3.9.5 Similarity Transformations
				Orthogonally and Unitarily Similar Transformations
				Uses of Similarity Transformations
			3.9.6 Schur Factorization
			3.9.7 Similar Canonical Factorization: Diagonalizable Matrices
				Symmetric Matrices
				A Defective Matrix
				The Jordan Decomposition
			3.9.8 Properties of Diagonalizable Matrices
				Matrix Functions
			3.9.9 Eigenanalysis of Symmetric Matrices
				Orthogonality of Eigenvectors: Orthogonal Diagonalization
				Spectral Decomposition
				Kronecker Products of Symmetric Matrices: Orthogonal Diagonalization
				Quadratic Forms and the Rayleigh Quotient
				The Fourier Expansion
				Powers of a Symmetric Matrix
				The Trace and the Sum of the Eigenvalues
			3.9.10 Generalized Eigenvalues and Eigenvectors
				Matrix Pencils
			3.9.11 Singular Values and the Singular Value Decomposition (SVD)
				The Fourier Expansion in Terms of the Singular Value Decomposition
				The Singular Value Decomposition and the Orthogonally Diagonal Factorization
		3.10 Positive Definite and Nonnegative Definite Matrices
			3.10.1 Eigenvalues of Positive and Nonnegative Definite Matrices
			3.10.2 Inverse of Positive Definite Matrices
			3.10.3 Diagonalization of Positive Definite Matrices
			3.10.4 Square Roots of Positive and Nonnegative Definite Matrices
		3.11 Matrix Norms
			3.11.1 Matrix Norms Induced from Vector Norms
				Lp Matrix Norms
				L1, L2, and L∞ Norms of Symmetric Matrices
			3.11.2 The Frobenius Norm—The ``Usual\'\' Norm
				The Frobenius Norm and the Singular Values
			3.11.3 Other Matrix Norms
			3.11.4 Matrix Norm Inequalities
			3.11.5 The Spectral Radius
			3.11.6 Convergence of a Matrix Power Series
				Conditions for Convergence of a Sequence of Powers to 0
				Another Perspective on the Spectral Radius: Relation to Norms
				Convergence of a Power Series: Inverse of I-A
				Nilpotent Matrices
		3.12 Approximation of Matrices
			3.12.1 Measures of the Difference between Two Matrices
			3.12.2 Best Approximation with a Matrix of Given Rank
		Appendix: Matrices in R
		Exercises
	4 Matrix Transformations and Factorizations
		Factorizations
		Computational Methods: Direct and Iterative
		4.1 Linear Geometric Transformations
			4.1.1 Invariance Properties of Linear Transformations
			4.1.2 Transformations by Orthogonal Matrices
			4.1.3 Rotations
			4.1.4 Reflections
			4.1.5 Translations; Homogeneous Coordinates
		4.2 Householder Transformations (Reflections)
			4.2.1 Zeroing All Elements but One in a Vector
			4.2.2 Computational Considerations
		4.3 Givens Transformations (Rotations)
			4.3.1 Zeroing One Element in a Vector
			4.3.2 Givens Rotations That Preserve Symmetry
			4.3.3 Givens Rotations to Transform to Other Values
			4.3.4 Fast Givens Rotations
		4.4 Factorization of Matrices
		4.5 LU and LDU Factorizations
			4.5.1 Properties: Existence
			4.5.2 Pivoting
			4.5.3 Use of Inner Products
			4.5.4 Properties: Uniqueness
			4.5.5 Properties of the LDU Factorization of a Square Matrix
		4.6 QR Factorization
			4.6.1 Related Matrix Factorizations
			4.6.2 Matrices of Full Column Rank
				Relation to the Moore-Penrose Inverse for Matrices of Full Column Rank
			4.6.3 Nonfull Rank Matrices
				Relation to the Moore-Penrose Inverse
			4.6.4 Determining the Rank of a Matrix
			4.6.5 Formation of the QR Factorization
			4.6.6 Householder Reflections to Form the QR Factorization
			4.6.7 Givens Rotations to Form the QR Factorization
			4.6.8 Gram-Schmidt Transformations to Form the  QR Factorization
		4.7 Factorizations of Nonnegative Definite Matrices
			4.7.1 Square Roots
			4.7.2 Cholesky Factorization
				Cholesky Decomposition of Singular Nonnegative Definite Matrices
				Relations to Other Factorizations
			4.7.3 Factorizations of a Gramian Matrix
		4.8 Approximate Matrix Factorization
			4.8.1 Nonnegative Matrix Factorization
			4.8.2 Incomplete Factorizations
		Appendix: R Functions for Matrix Computations and for Graphics
		Exercises
	5 Solution of Linear Systems
		5.1 Condition of Matrices
			5.1.1 Condition Number
			5.1.2 Improving the Condition Number
				Ridge Regression and the Condition Number
			5.1.3 Numerical Accuracy
		5.2 Direct Methods for Consistent Systems
			5.2.1 Gaussian Elimination and Matrix Factorizations
				Pivoting
				Nonfull Rank and Nonsquare Systems
			5.2.2 Choice of Direct Method
		5.3 Iterative Methods for Consistent Systems
			5.3.1 The Gauss-Seidel Method with Successive Overrelaxation
				Convergence of the Gauss-Seidel Method
				Successive Overrelaxation
			5.3.2 Conjugate Gradient Methods for Symmetric Positive Definite Systems
				The Conjugate Gradient Method
				Krylov Methods
				GMRES Methods
				Preconditioning
			5.3.3 Multigrid Methods
		5.4 Iterative Refinement
		5.5 Updating a Solution to a Consistent System
		5.6 Overdetermined Systems: Least Squares
			5.6.1 Least Squares Solution of an Overdetermined System
				Orthogonality of Least Squares Residuals to span(X)
				Numerical Accuracy in Overdetermined Systems
			5.6.2 Least Squares with a Full Rank Coefficient Matrix
			5.6.3 Least Squares with a Coefficient Matrix Not of Full Rank
				An Optimal Property of the Solution Using the Moore-Penrose Inverse
			5.6.4 Weighted Least Squares
			5.6.5 Updating a Least Squares Solution of an Overdetermined System
		5.7 Other Solutions of Overdetermined Systems
			5.7.1 Solutions That Minimize Other Norms of the Residuals
				Minimum L1 Norm Fitting: Least Absolute Values
				Minimum L∞ Norm Fitting: Minimax
				Lp Norms and Iteratively Reweighted Least Squares
			5.7.2 Regularized Solutions
			5.7.3 Other Restrictions on the Solutions
			5.7.4 Minimizing Orthogonal Distances
		Appendix: R Functions for Solving Linear Systems
		Exercises
	6 Evaluation of Eigenvalues and Eigenvectors
		6.1 General Computational Methods
			6.1.1 Numerical Condition of an Eigenvalue Problem
			6.1.2 Eigenvalues from Eigenvectors and Vice Versa
			6.1.3 Deflation
				Deflation of Symmetric Matrices
			6.1.4 Preconditioning
			6.1.5 Shifting
		6.2 Power Method
		6.3 Jacobi Method
		6.4 QR Method
		6.5 Krylov Methods
		6.6 Generalized Eigenvalues
		6.7 Singular Value Decomposition
		Exercises
	7 Real Analysis and Probability Distributions of Vectors and Matrices
		7.1 Basics of Differentiation
			7.1.1 Continuity
			7.1.2 Notation and Properties
			7.1.3 Differentials
			7.1.4 Use of Differentiation in Optimization
		7.2 Types of Differentiation
			7.2.1 Differentiation with Respect to a Scalar
				Derivatives of Vectors with Respect to Scalars
				Derivatives of Matrices with Respect to Scalars
				Derivatives of Functions with Respect to Scalars
				Higher-Order Derivatives with Respect to Scalars
			7.2.2 Differentiation with Respect to a Vector
				Derivatives of Scalars with Respect to Vectors; The Gradient
				Derivatives of Vectors with Respect to Vectors; The Jacobian
				Derivatives of Vectors with Respect to Vectors in IR3; The Divergence and the Curl
				Derivatives of Matrices with Respect to Vectors
				Higher-Order Derivatives with Respect to Vectors; The Hessian
				Summary of Derivatives with Respect to Vectors
			7.2.3 Differentiation with Respect to a Matrix
		7.3 Integration
			7.3.1 Multidimensional Integrals and Integrals Involving Vectors and Matrices
			7.3.2 Change of Variables; The Jacobian
			7.3.3 Integration Combined with Other Operations
		7.4 Multivariate Probability Theory
			7.4.1 Random Variables and Probability Distributions
				The Distribution Function and Probability Density Function
				Expected Values; The Expectation Operator
				Expected Values; Generating Functions
				Vector Random Variables
				Matrix Random Variables
				Special Random Variables
			7.4.2 Distributions of Transformations of Random Variables
				Change-of-Variables Method
				Inverse CDF Method
				Moment-Generating Function Method
			7.4.3 The Multivariate Normal Distribution
				Linear Transformations of a Multivariate Normal Random Variable
				The Matrix Normal Distribution
			7.4.4 Distributions Derived from the Multivariate Normal
			7.4.5 Chi-Squared Distributions
				The Family of Distributions Nn(0,σ2In)
				The Family of Distributions Nd(μ,Σ)
			7.4.6 Wishart Distributions
		7.5 Multivariate Random Number Generation
			7.5.1 The Multivariate Normal Distribution
			7.5.2 Random Correlation Matrices
		Appendix: R for Working with Probability Distributions and for Simulating Random Data
		Exercises
Part II Applications in Statistics and Data Science
	8 Special Matrices and Operations Useful in Modeling and Data Science
		8.1 Data Matrices and Association Matrices
			8.1.1 Flat Files
			8.1.2 Graphs and Other Data Structures
				Adjacency Matrix: Connectivity Matrix
				Digraphs
				Connectivity of Digraphs
				Irreducible Matrices
				Strong Connectivity of Digraphs and Irreducibility of Matrices
			8.1.3 Term-by-Document Matrices
			8.1.4 Sparse Matrices
			8.1.5 Probability Distribution Models
			8.1.6 Derived Association Matrices
		8.2 Symmetric Matrices and Other Unitarily Diagonalizable Matrices
			8.2.1 Some Important Properties of Symmetric Matrices
			8.2.2 Approximation of Symmetric Matrices and an Important Inequality
			8.2.3 Normal Matrices
		8.3 Nonnegative Definite Matrices: Cholesky Factorization
			8.3.1 Eigenvalues of Nonnegative Definite Matrices
			8.3.2 The Square Root and the Cholesky Factorization
			8.3.3 The Convex Cone of Nonnegative Definite Matrices
		8.4 Positive Definite Matrices
			8.4.1 Leading Principal Submatrices of Positive Definite Matrices
			8.4.2 The Convex Cone of Positive Definite Matrices
			8.4.3 Inequalities Involving Positive Definite Matrices
		8.5 Idempotent and Projection Matrices
			8.5.1 Idempotent Matrices
				Symmetric Idempotent Matrices
				Cochran\'s Theorem
			8.5.2 Projection Matrices: Symmetric Idempotent Matrices
				Projections onto Linear Combinations of Vectors
		8.6 Special Matrices Occurring in Data Analysis
			8.6.1 Gramian Matrices
				Sums of Squares and Cross-Products
				Some Immediate Properties of Gramian Matrices
				Generalized Inverses of Gramian Matrices
				Eigenvalues of Gramian Matrices
			8.6.2 Projection and Smoothing Matrices
				A Projection Matrix Formed from a Gramian Matrix
				Smoothing Matrices
				Effective Degrees of Freedom
				Residuals from Smoothed Data
			8.6.3 Centered Matrices and Variance-Covariance Matrices
				Centering and Scaling of Data Matrices
				Gramian Matrices Formed from Centered Matrices: Covariance Matrices
				Gramian Matrices Formed from Scaled Centered Matrices: Correlation Matrices
			8.6.4 The Generalized Variance
				Comparing Variance-Covariance Matrices
			8.6.5 Similarity Matrices
			8.6.6 Dissimilarity Matrices
		8.7 Nonnegative and Positive Matrices
			The Convex Cones of Nonnegative and Positive Matrices
			8.7.1 Properties of Square Positive Matrices
			8.7.2 Irreducible Square Nonnegative Matrices
				Properties of Square Irreducible Nonnegative Matrices; the Perron-Frobenius Theorem
				Primitive Matrices
				Limiting Behavior of Primitive Matrices
			8.7.3 Stochastic Matrices
			8.7.4 Leslie Matrices
		8.8 Other Matrices with Special Structures
			8.8.1 Helmert Matrices
			8.8.2 Vandermonde Matrices
			8.8.3 Hadamard Matrices and Orthogonal Arrays
			8.8.4 Toeplitz Matrices
				Inverses of Certain Toeplitz Matrices and Other Banded Matrices
			8.8.5 Circulant Matrices
			8.8.6 Fourier Matrices and the Discrete Fourier Transform
				Fourier Matrices and Elementary Circulant Matrices
				The Discrete Fourier Transform
			8.8.7 Hankel Matrices
			8.8.8 Cauchy Matrices
			8.8.9 Matrices Useful in Graph Theory
				Adjacency Matrix: Connectivity Matrix
				Digraphs
				Use of the Connectivity Matrix
				The Laplacian Matrix of a Graph
			8.8.10 Z-Matrices and M-Matrices
		Exercises
	9 Selected Applications in Statistics
		Structure in Data and Statistical Data Analysis
		9.1 Linear Models
			Notation
			Statistical Inference
			9.1.1 Fitting the Model
				Ordinary Least Squares
				Weighted Least Squares
				Variations on the Criteria for Fitting
				Regularized Fits
				Orthogonal Distances
				Collinearity
			9.1.2 Least Squares Fit of Full-Rank Models
			9.1.3 Least Squares Fits of Nonfull-Rank Models
				A Classification Model: Numerical Example
				Fitting the Model Using Generalized Inverses
				Uniqueness
			9.1.4 Computing the Solution
				Direct Computations on X
				The Normal Equations and the Sweep Operator
				Computations for Analysis of Variance
			9.1.5 Properties of a Least Squares Fit
				Geometrical Properties
				Degrees of Freedom
				The Hat Matrix and Leverage
			9.1.6 Linear Least Squares Subject to Linear Equality Constraints
			9.1.7 Weighted Least Squares
			9.1.8 Updating Linear Regression Statistics
				Adding More Variables
				Adding More Observations
				Adding More Observations Using Weights
			9.1.9 Linear Smoothing
			9.1.10 Multivariate Linear Models
				Fitting the Model
				Partitioning the Sum of Squares
		9.2 Statistical Inference in Linear Models
			Statistical Properties of Estimators
			Full-Rank and Nonfull-Rank Models
			9.2.1 The Probability Distribution of ε
				Expectation of ε
				Variances of ε and of the Least Squares Fits
				Normality: εNn(0,σ2In)
				Maximum Likelihood Estimators
			9.2.2 Estimability
				Uniqueness and Unbiasedness of Least Squares Estimators
				Variance of Least Squares Estimators of Estimable Combinations
				The Classification Model Numerical Example (Continued from Page ??
			9.2.3 The Gauss-Markov Theorem
			9.2.4 Testing Linear Hypotheses
			9.2.5 Statistical Inference in Linear Models with Heteroscedastic or Correlated Errors
			9.2.6 Statistical Inference for Multivariate Linear Models
		9.3 Principal Components
			9.3.1 Principal Components of a Random Vector
			9.3.2 Principal Components of Data
				Principal Components Directly from the Data Matrix
				Dimension Reduction
		9.4 Condition of Models and Data
			9.4.1 Ill-Conditioning in Statistical Applications
			9.4.2 Variable Selection
			9.4.3 Principal Components Regression
			9.4.4 Shrinkage Estimation
				Ridge Regression
				Lasso Regression
				Elastic Net
			9.4.5 Statistical Inference About the Rank of a Matrix
				Numerical Approximation and Statistical Inference
				Statistical Tests of the Rank of a Class of Matrices
				Statistical Tests of the Rank Based on an LDU Factorization
			9.4.6 Incomplete Data
		9.5 Stochastic Processes
			9.5.1 Markov Chains
				Properties of Markov Chains
				Limiting Behavior of Markov Chains
			9.5.2 Markovian Population Models
			9.5.3 Autoregressive Processes
				Relation of the Autocorrelations to the Autoregressive Coefficients
		9.6 Optimization of Scalar-Valued Functions
			9.6.1 Stationary Points of Functions
			9.6.2 Newton\'s Method
				Quasi-Newton Methods
			9.6.3 Least Squares
				Linear Least Squares
				Nonlinear Least Squares: The Gauss-Newton Method
				Levenberg-Marquardt Method
			9.6.4 Maximum Likelihood
			9.6.5 Optimization of Functions with Constraints
				Equality-Constrained Linear Least Squares Problems
				The Reduced Gradient and Reduced Hessian
				Lagrange Multipliers
				The Lagrangian
				Another Example: The Rayleigh Quotient
				Optimization of Functions with Inequality Constraints
				Inequality-Constrained Linear Least Squares Problems
				Nonlinear Least Squares as an Inequality-Constrained Problem
			9.6.6 Optimization Without Differentiation
		Appendix: R for Applications in Statistics
		Exercises
Part III Numerical Methods and Software
	10 Numerical Methods
		10.1 Software Development
			10.1.1 Standards
			10.1.2 Coding Systems
			10.1.3 Types of Data
			10.1.4 Missing Data
			10.1.5 Data Structures
			10.1.6 Computer Architectures and File Systems
		10.2 Digital Representation of Numeric Data
			10.2.1 The Fixed-Point Number System
				Software Representation and Big Integers
			10.2.2 The Floating-Point Model for Real Numbers
				The Parameters of the Floating-Point Representation
				Standardization of Floating-Point Representation
				Special Floating-Point Numbers
			10.2.3 Language Constructs for Representing Numeric Data
				C
				Fortran
				Determining the Numerical Characteristics of a Particular Computer
			10.2.4 Other Variations in the Representation of Data: Portability of Data
		10.3 Computer Operations on Numeric Data
			10.3.1 Fixed-Point Operations
			10.3.2 Floating-Point Operations
				Errors
				Guard Digits and Chained Operations
				Addition of Several Numbers
				Compensated Summation
				Catastrophic Cancellation
				Standards for Floating-Point Operations
				Operations Involving Special Floating-Point Numbers
				Comparison of Real Numbers and Floating-Point Numbers
			10.3.3 Language Constructs for Operations on Numeric Data
			10.3.4 Software Methods for Extending the Precision
				Multiple Precision
				Rational Fractions
				Interval Arithmetic
			10.3.5 Exact Computations
				Exact Dot Product (EDP)
		10.4 Numerical Algorithms and Analysis
			Algorithms and Programs
			10.4.1 Error in Numerical Computations
				Measures of Error and Bounds for Errors
				Error of Approximation
				Algorithms and Data
				Condition of Data
				Robustness of Algorithms
				Stability of Algorithms
				Reducing the Error in Numerical Computations
			10.4.2 Efficiency
				Measuring Efficiency: Counting Computations
				Measuring Efficiency: Timing Computations
				Improving Efficiency
				Scalability
				Bottlenecks and Limits
				High-Performance Computing
				Computations in Parallel
			10.4.3 Iterations and Convergence
				Extrapolation
			10.4.4 Computations Without Storing Data
			10.4.5 Other Computational Techniques
				Recursion
				MapReduce
		Appendix: Numerical Computations in R
		Exercises
	11 Numerical Linear Algebra
		11.1 Computer Storage of Vectors and Matrices
			11.1.1 Storage Modes
			11.1.2 Strides
			11.1.3 Sparsity
		11.2 General Computational Considerations for Vectors and Matrices
			11.2.1 Relative Magnitudes of Operands
				Condition
				Pivoting
				``Modified\'\' and ``Classical\'\' Gram-Schmidt Transformations
			11.2.2 Iterative Methods
				Preconditioning
				Restarting and Rescaling
				Preservation of Sparsity
				Iterative Refinement
			11.2.3 Assessing Computational Errors
				Assessing Errors in Given Computations
		11.3 Multiplication of Vectors and Matrices
			11.3.1 Strassen\'s Algorithm
			11.3.2 Matrix Multiplication Using MapReduce
		11.4 Other Matrix Computations
			11.4.1 Rank Determination
			11.4.2 Computing the Determinant
			11.4.3 Computing the Condition Number
		Exercises
	12 Software for Numerical Linear Algebra
		12.1 General Considerations
			12.1.1 Software Development and Open-Source Software
			12.1.2 Integrated Development, Collaborative Research, and Version Control
			12.1.3 Finding Software
			12.1.4 Software Design
				Interoperability
				Efficiency
				Writing Mathematics and Writing Programs
				Numerical Mathematical Objects and Computer Objects
				Other Mathematical Objects and Computer Objects
				Software for Statistical Applications
				Robustness
				Computing Paradigms: Parallel Processing
				Array Structures and Indexes
				Matrix Storage Modes
				Storage Schemes for Sparse Matrices
			12.1.5 Software Development, Maintenance, and Testing
				Test Data
				Assessing the Accuracy of a Computed Result
				Software Reviews
			12.1.6 Reproducible Research
		12.2 Software Libraries
			12.2.1 BLAS
			12.2.2 Level 2 and Level 3 BLAS, LAPACK, and Related Libraries
			12.2.3 Libraries for High-Performance Computing
				Libraries for Parallel Processing
				Parallel Computations in R
				Graphical Processing Units
				Clusters of Computers and Cloud Computing
			12.2.4 The IMSL Libraries
				Examples of Use of the IMSL Libraries
		12.3 General-Purpose Languages and Programming Systems
			12.3.1 Programming Considerations
				Reverse Communication in Iterative Algorithms
				Computational Efficiency
			12.3.2 Modern Fortran
			12.3.3 C and C++
			12.3.4 Python
			12.3.5 MATLAB and Octave
		Appendix: R Software for Numerical Linear Algebra
		Exercises
Appendices
	A Notation and Definitions
		A.1 General Notation
		A.2 Computer Number Systems
		A.3 General Mathematical Functions and Operators
		A.4 Linear Spaces and Matrices
			A.4.1 Norms and Inner Products
			A.4.2 Matrix Shaping Notation
			A.4.3 Notation for Rows or Columns of Matrices
			A.4.4 Notation Relating to Matrix Determinants
			A.4.5 Matrix-Vector Differentiation
			A.4.6 Special Vectors and Matrices
			A.4.7 Elementary Operator Matrices
		A.5 Models and Data
	Bibliography
	Index