Applied Multivariate Statistical Analysis

Preface
Contents
Part I Descriptive Techniques
	1 Comparison of Batches
		1.1 Boxplots
			1.1.1 Construction of the Boxplot
		1.2 Histograms
		1.3 Kernel Densities
		1.4 Scatterplots
		1.5 Chernoff-Flury Faces
		1.6 Andrews\' Curves
		1.7 Parallel Coordinate Plots
		1.8 Hexagon Plots
		1.9 Boston Housing
			1.9.1 Aim of the Analysis
			1.9.2 What Can Be Seen from the PCPs
			1.9.3 The Scatterplot Matrix
				1.9.3.1 Per Capita Crime Rate X1
				1.9.3.2 Proportion of Residential Area Zoned for Large Lots X2
				1.9.3.3 Proportion of Non-retail Business Acres X3
				1.9.3.4 Charles River Dummy Variable X4
				1.9.3.5 Nitric Oxide Concentration X5
				1.9.3.6 Average Number of Rooms per Dwelling X6
				1.9.3.7 Proportion of Owner-Occupied Units Built Prior to 1940 X7
				1.9.3.8 Weighted Distance to Five Boston Employment Centers X8
				1.9.3.9 Index of Accessibility to Radial Highways X9
				1.9.3.10 Full-Value Property Tax X10
				1.9.3.11 Pupil/Teacher Ratio X11
				1.9.3.12 Proportion of African American B, X12 = 1000 (B - 0.63)2 I(B<0.63)
				1.9.3.13 Proportion of Lower Status of the Population X13
			1.9.4 Transformations
		1.10 Exercises
		References
Part II Multivariate Random Variables
	2 A Short Excursion into Matrix Algebra
		2.1 Elementary Operations
			2.1.1 Matrix Operations
			2.1.2 Properties of Matrix Operations
			2.1.3 Matrix Characteristics
			2.1.4 Rank
			2.1.5 Trace
			2.1.6 Determinant
			2.1.7 Transpose
			2.1.8 Inverse
			2.1.9 G-inverse
			2.1.10 Eigenvalues, Eigenvectors
			2.1.11 Properties of Matrix Characteristics
		2.2 Spectral Decompositions
		2.3 Quadratic Forms
			2.3.1 Definiteness of Quadratic Forms and Matrices
		2.4 Derivatives
		2.5 Partitioned Matrices
		2.6 Geometrical Aspects
			2.6.1 Distance
			2.6.2 Remark: Usefulness of Theorem 2.7
			2.6.3 Norm of a Vector
			2.6.4 Angle Between Two Vectors
			2.6.5 Rotations
			2.6.6 Column Space and Null Space of a Matrix
			2.6.7 Projection Matrix
			2.6.8 Projection on C(X)
		2.7 Exercises
	3 Moving to Higher Dimensions
		3.1 Covariance
		3.2 Correlation
		3.3 Summary Statistics
			3.3.1 Linear Transformation
			3.3.2 Mahalanobis Transformation
		3.4 Linear Model for Two Variables
		3.5 Simple Analysis of Variance
			3.5.1 The F-Test in a Linear Regression Model
		3.6 Multiple Linear Model
			3.6.1 Properties of β\"0362β
			3.6.2 The ANOVA Model in Matrix Notation
		3.7 Boston Housing
		3.8 Exercises
		References
	4 Multivariate Distributions
		4.1 Distribution and Density Function
		4.2 Moments and Characteristic Functions
			4.2.1 Moments: Expectation and Covariance Matrix
			4.2.2 Properties of the Covariance Matrix Σ=Var(X)
			4.2.3 Properties of Variances and Covariances
			4.2.4 Conditional Expectations
			4.2.5 Properties of Conditional Expectations
			4.2.6 Characteristic Functions
			4.2.7 Cumulant Functions
		4.3 Transformations
		4.4 The Multinormal Distribution
			4.4.1 Geometry of the Np(μ, Σ) Distribution
			4.4.2 Singular Normal Distribution
			4.4.3 Gaussian Copula
		4.5 Sampling Distributions and Limit Theorems
			4.5.1 Transformation of Statistics
		4.6 Heavy-Tailed Distributions
			4.6.1 Generalized Hyperbolic Distribution
			4.6.2 Student\'s T-Distribution
			4.6.3 Laplace Distribution
			4.6.4 Cauchy Distribution
			4.6.5 Mixture Model
			4.6.6 Multivariate Generalized Hyperbolic Distribution
			4.6.7 Multivariate T-Distribution
			4.6.8 Multivariate Laplace Distribution
			4.6.9 Multivariate Mixture Model
			4.6.10 Generalized Hyperbolic Distribution
		4.7 Copulae
		4.8 Bootstrap
		4.9 Exercises
		References
	5 Theory of the Multinormal
		5.1 Elementary Properties of the Multinormal
			5.1.1 Conditional Approximations
		5.2 The Wishart Distribution
		5.3 Hotelling\'s T2-Distribution
		5.4 Spherical and Elliptical Distributions
		5.5 Exercises
		References
	6 Theory of Estimation
		6.1 The Likelihood Function
		6.2 The Cramer-Rao Lower Bound
		6.3 Exercises
		Reference
	7 Hypothesis Testing
		7.1 Likelihood Ratio Test
			7.1.1 Testing for the Mean
			7.1.2 Confidence Region for μ
		7.2 Linear Hypothesis
			7.2.1 General Framework
			7.2.2 Repeated Measurements
			7.2.3 Comparison of Two Mean Vectors
			7.2.4 Profile Analysis
				7.2.4.1 Parallel Profiles
				7.2.4.2 Equality of Two Levels
				7.2.4.3 Treatment Effect
		7.3 Boston Housing
			7.3.1 Testing the Equality in Means
			7.3.2 Testing Linear Restrictions
		7.4 Exercises
		References
Part III Multivariate Techniques
	8 Regression Models
		8.1 General ANOVA and ANCOVA Models
			8.1.1 ANOVA Models
			8.1.2 ANCOVA Models
			8.1.3 Boston Housing
		8.2 Categorical Responses
			8.2.1 Multinomial Sampling and Contingency Tables
			8.2.2 Log-Linear Models for Contingency Tables
				8.2.2.1 Two-Way Tables
				8.2.2.2 Three-Way Tables
			8.2.3 Testing Issues with Count Data
			8.2.4 Logit Models
				8.2.4.1 Logit Models for Binary Response
				8.2.4.2 Logit Models for Contingency Tables
		8.3 Exercises
		Reference
	9 Variable Selection
		9.1 Lasso
			9.1.1 Lasso in the Linear Regression Model
				9.1.1.1 Geometrical Aspects in R2
				9.1.1.2 The LAR Algorithm and Lasso Solution Paths
				9.1.1.3 Model Selection
				9.1.1.4 Orthonormal Design Case
				9.1.1.5 General Lasso Solutions
			9.1.2 Lasso in High Dimensions
			9.1.3 Lasso in Logit Model
		9.2 Elastic Net
			9.2.1 Elastic Net in the Linear Regression Model
			9.2.2 Elastic Net in the Logit Model
		9.3 Group Lasso
		9.4 Exercises
		References
	10 Decomposition of Data Matrices by Factors
		10.1 The Geometric Point of View
		10.2 Fitting the p-Dimensional Point Cloud
			10.2.1 Subspaces of Dimension 1
			10.2.2 Representation of the Cloud on  F1
			10.2.3 Subspaces of Dimension 2
			10.2.4 Subspaces of Dimension q (q≤p)
		10.3 Fitting the n-Dimensional Point Cloud
			10.3.1 Subspaces of Dimension 1
			10.3.2 Representation of the Cloud on G1
			10.3.3 Subspaces of Dimension q (q≤n)
		10.4 Relations Between Subspaces
		10.5 Practical Computation
		10.6 Exercises
	11 Principal Component Analysis
		11.1 Standardized Linear Combination
		11.2 Principal Components in Practice
		11.3 Interpretation of the PCs
		11.4 Asymptotic Properties of the PCs
			11.4.1 Variance Explained by the First q PCs
		11.5 Normalized Principal Component Analysis
		11.6 Principal Components as a Factorial Method
			11.6.1 Quality of the Representations
		11.7 Common Principal Components
		11.8 Boston Housing
		11.9 More Examples
		11.10 Exercises
		References
	12 Factor Analysis
		12.1 The Orthogonal Factor Model
			12.1.1 Interpretation of the Factors
			12.1.2 Invariance of Scale
			12.1.3 Nonuniqueness of Factor Loadings
		12.2 Estimation of the Factor Model
			12.2.1 The Maximum Likelihood Method
				12.2.1.1 Likelihood Ratio Test for the Number of Common Factors
			12.2.2 The Method of Principal Factors
			12.2.3 The Principal Component Method
			12.2.4 Rotation
		12.3 Factor Scores and Strategies
			12.3.1 Practical Suggestions
			12.3.2 Factor Analysis Versus PCA
		12.4 Boston Housing
		12.5 Exercises
		References
	13 Cluster Analysis
		13.1 The Problem
		13.2 The Proximity Between Objects
			13.2.1 Similarity of Objects with Binary Structure
			13.2.2 Distance Measures for Continuous Variables
		13.3 Cluster Algorithms
			13.3.1 Partitioning Algorithms
			13.3.2 Hierarchical Algorithms, Agglomerative Techniques
		13.4 Adaptive Weights Clustering
			13.4.1 Sequence of Radii
			13.4.2 Initialization of Weights
			13.4.3 Updates at Step k
			13.4.4 Parameter Tuning
		13.5 Spectral Clustering
			13.5.1 Relation to the Graph Cut Problem
		13.6 Boston Housing
		13.7 Exercises
		References
	14 Discriminant Analysis
		14.1 Allocation Rules for Known Distributions
			14.1.1 Maximum Likelihood Discriminant Rule
			14.1.2 Bayes Discriminant Rule
			14.1.3 Probability of Misclassification for the ML Rule (J=2)
			14.1.4 Classification with Different Covariance Matrices
		14.2 Discrimination Rules in Practice
			14.2.1 Estimation of the Probabilities of Misclassifications
			14.2.2 Fisher\'s Linear Discrimination Function
		14.3 Boston Housing
		14.4 Exercises
		References
	15 Correspondence Analysis
		15.1 Motivation
		15.2 Chi-Square Decomposition
		15.3 Correspondence Analysis in Practice
			15.3.1 Biplots
		15.4 Exercises
	16 Canonical Correlation Analysis
		16.1 Most Interesting Linear Combination
		16.2 Canonical Correlation in Practice
			16.2.1 Testing the Canonical Correlation Coefficients
			16.2.2 Canonical Correlation Analysis with Qualitative Data
		16.3 Exercises
		References
	17 Multidimensional Scaling
		17.1 The Problem
		17.2 Metric Multidimensional Scaling
			17.2.1 The Classical Solution
				17.2.1.1 Recovery of Coordinates
				17.2.1.2 How Many Dimensions?
				17.2.1.3 Similarities
				17.2.1.4 Relation to Factorial Analysis
				17.2.1.5 Optimality Properties of the Classical MDS Solution
		17.3 Nonmetric Multidimensional Scaling
			17.3.1 Shepard-Kruskal Algorithm
		17.4 Exercises
		References
	18 Conjoint Measurement Analysis
		18.1 Introduction
		18.2 Design of Data Generation
		18.3 Estimation of Preference Orderings
			18.3.1 Metric Solution
			18.3.2 Nonmetric Solution
		18.4 Exercises
		References
	19 Applications in Finance
		19.1 Portfolio Choice
		19.2 Efficient Portfolio
			19.2.1 Nonexistence of a Riskless Asset
			19.2.2 Existence of a Riskless Asset
		19.3 Efficient Portfolios in Practice
		19.4 The Capital Asset Pricing Model (CAPM)
		19.5 Exercises
		Reference
	20 Computationally Intensive Techniques
		20.1 Simplicial Depth
		20.2 Projection Pursuit
			20.2.1 Exploratory Projection Pursuit
			20.2.2 Projection Pursuit Regression
		20.3 Sliced Inverse Regression
			20.3.1 The SIR Algorithm
			20.3.2 SIR II
			20.3.3 The SIR II Algorithm
		20.4 Support Vector Machines
			20.4.1 Classification Methodology
			20.4.2 Expected vs. Empirical Risk Minimization
			20.4.3 The SVM in the Linearly Separable Case
			20.4.4 SVMs in the Linearly Nonseparable Case
			20.4.5 Nonlinear Classification
			20.4.6 SVMs for Simulated Data
			20.4.7 Solution of the SVM Classification Problem
			20.4.8 Scoring Companies
		20.5 Classification and Regression Trees
			20.5.1 How Does CART Work?
			20.5.2 Impurity Measures
			20.5.3 Gini Index and Twoing Rule in Practice
			20.5.4 Optimal Size of a Decision Tree
			20.5.5 Cross-Validation for Tree Pruning
			20.5.6 Cost-Complexity Function and Cross-Validation
			20.5.7 Regression Trees
			20.5.8 Bankruptcy Analysis
		20.6 Boston Housing
		20.7 Exercises
		References
	21 Locally Linear Embedding
		21.1 Introduction
		21.2 The Basic Ideas of LLE
		21.3 The LLE Algorithm Step by Step
			21.3.1 Step 1: k-Nearest Neighbors
			21.3.2 Step 2: Linear Approximation by the NearestNeighbors
			21.3.3 Step 3: The Linear Embedding
			21.3.4 Graph-Theoretic Interpretation of the LLE Solution
		21.4 Swiss Roll Data
		21.5 Boston Housing Data
		21.6 Exercises
		References
	22 Stochastic Neighborhood Embedding
		22.1 Introduction
		22.2 Modeling Neighborhood
			22.2.1 The Input Space Relies on the Gaussian Law
			22.2.2 The Embedding Space Employs the t-Distribution
		22.3 Finding the t-SNE Embedding
		22.4 Applications of t-SNE
			22.4.1 The Trefoil Knot
			22.4.2 Quantlet Clustering
		22.5 Exercises
		References
	23 Uniform Manifold Approximation and Projection
		23.1 Introduction
		23.2 The Basic Ideas of UMAP
		23.3 The Computational Details of UMAP
			23.3.1 The Fuzzy Topological Representation in the Input Space
			23.3.2 Representation in the Embedding Space
			23.3.3 Computing the Embedding
		23.4 Examples
			23.4.1 Banknotes Data
			23.4.2 The Trefoil Knot
		23.5 Exercises
		References
A Symbols and Notations
	Basics
	Mathematical Abbreviations
	Samples
	Densities and Distribution Functions
	Moments
	Empirical Moments
	Distributions
B Data
	B.1 Boston Housing Data
	B.2 Swiss Bank Notes
	B.3 Car Data
	B.4 Classic Blue Pullovers Data
	B.5 U.S. Companies Data
	B.6 French Food Data
	B.7 Car Marks
	B.8 U.S. Crime Data
	B.9 Bankruptcy Data I
	B.10 Bankruptcy Data II
	B.11 Journaux Data
	B.12 Timebudget Data
	B.13 Vocabulary Data
	B.14 French Baccalauréat Frequencies
	References
Index