Matrix-Based Introduction To Multivariate Data Analysis

Preface to the Second Edition......Page 5
Preface to the First Edition......Page 8
Contents......Page 11
Elementary Statistics with Matrices......Page 18
1.1 Matrices......Page 19
1.2 Vectors......Page 21
1.3 Sum of Matrices and Their Multiplication by Scalars......Page 22
1.4 Inner Product and Norms of Vectors......Page 23
1.5 Product of Matrices......Page 24
1.7 Trace Operator and Matrix Norm......Page 27
1.8 Vectors and Matrices Filled with Ones or Zeros......Page 29
1.9 Special Square Matrices......Page 30
1.10 Bibliographical Notes......Page 32
2.1 Data Matrices......Page 33
2.2 Distributions......Page 35
2.4 Centered Scores......Page 36
2.5 Variances and Standard Deviations......Page 39
2.6 Standard Scores......Page 41
2.7 What Centering and Standardization Do for Distributions......Page 42
2.8 Matrix Representation......Page 43
2.9 Bibliographical Notes......Page 44
3.1 Scatter Plots and Correlations......Page 46
3.2 Covariances......Page 47
3.3 Correlation Coefficients......Page 49
3.4 Variable Vectors and Correlations......Page 51
3.5 Covariances and Correlations for Standard Scores......Page 52
3.6 Matrix Expressions of Covariances and Correlations......Page 53
3.7 Unbiased Covariances......Page 54
3.8 Centered Matrices......Page 55
3.9 Ranks of Matrices: Intuitive Introduction......Page 56
3.10 Ranks of Matrices: Mathematical Definition......Page 57
3.11 Bibliographical Notes......Page 59
Least Squares Procedures......Page 61
4.1 Prediction of a Dependent Variable by Explanatory Variables......Page 62
4.2 Least Squares Method......Page 65
4.3 Predicted and Error Values......Page 67
4.4 Proportion of Explained Variance and Multiple Correlation......Page 69
4.5 Interpretation of Regression Coefficients......Page 71
4.6 Standardization......Page 73
4.7 Geometric Derivation of Regression Coefficients......Page 74
4.8 Bibliographical Notes......Page 75
5.1 Reduction of Variables into Components......Page 78
5.2 Singular Value Decomposition......Page 80
5.3 Formulation with a Weight Matrix......Page 83
5.4 Constraints for Components......Page 84
5.5 Interpretation of Loadings......Page 86
5.7 Percentage of Explained Variance......Page 87
5.8 High-Dimensional Data Analysis......Page 89
5.9 Bibliographical Notes......Page 90
6.1 Reformulation with Different Constraints......Page 94
6.2 Maximizing the Sum of Variances......Page 95
6.3 Weighted Composite Scores with Maximum Variance......Page 97
6.4 Projecting Three-Dimensional Vectors onto Two-Dimensional Ones......Page 99
6.5 Visualization of Invisible Distributions......Page 102
6.6 Goodness of Projection......Page 105
6.7 Bibliographical Notes......Page 106
7.1 Membership Matrices......Page 108
7.2 Example of Clustering Results......Page 109
7.3 Formulation......Page 111
7.4 Iterative Algorithm......Page 113
7.5 Obtaining Cluster Features......Page 115
7.6 Obtaining Memberships......Page 116
7.7 Brief Description of Algorithm......Page 117
7.8 Bibliographical Notes......Page 118
Maximum Likelihood Procedures......Page 121
8.1 Model, Parameter, Objective Function, and Optimization......Page 122
8.2 Maximum Likelihood Method......Page 123
8.3 Probability Density Function......Page 126
8.4 Multivariate Normal Distribution......Page 127
8.5 Maximum Likelihood Method for Normal Variables......Page 130
8.6 Maximum Likelihood Estimates of Means and Covariances......Page 132
8.7 Model Selection......Page 134
8.8 Assessment of Between-Group Heterogeneity......Page 135
8.9 Bibliographical Notes......Page 138
9.1 From Multiple Regression Analysis to Path Analysis......Page 142
9.2 Matrix Expression......Page 147
9.3 Distributional Assumptions......Page 148
9.4 Likelihood for Covariance Structure Analysis......Page 149
9.5 Maximum Likelihood Estimation......Page 150
9.6 Estimated Covariance Structure......Page 151
9.8 Other and Extreme Models......Page 153
9.9 Model Selection......Page 155
9.10 Bibliographical Notes......Page 156
10 Confirmatory Factor Analysis......Page 160
10.2 Matrix Expression......Page 161
10.3 Distributional Assumptions for Common Factors......Page 167
10.4 Distributional Assumptions for Errors......Page 168
10.5 Maximum Likelihood Method......Page 169
10.6 Solutions......Page 170
10.7 Other and Extreme Models......Page 171
10.8 Model Selection......Page 172
10.9 Bibliographical Notes......Page 173
11.1 Causality Among Factors......Page 175
11.2 Observed Variables as Indicator of Factors......Page 176
11.4 Matrix Expression......Page 177
11.5 Distributional Assumptions......Page 182
11.6 Maximum Likelihood Method......Page 183
11.7 Solutions......Page 184
11.8 Model Selection......Page 185
11.9 Bibliographical Notes......Page 186
12.1 Example of Exploratory Factor Analysis Model......Page 188
12.2 Matrix Expression......Page 189
12.3 Distributional Assumptions......Page 190
12.5 Indeterminacy of EFA Solutions......Page 191
12.7 Interpretation of Loadings......Page 193
12.9 Selecting the Number of Factors......Page 195
12.10 Difference to Principal Component Analysis......Page 197
12.11 Bibliographical Notes......Page 201
Miscellaneous Procedures......Page 204
13.1 Geometric Illustration of Factor Rotation......Page 205
13.3 Rotation to Simple Structure......Page 208
13.4 Varimax Rotation......Page 211
13.5 Geomin Rotation......Page 212
13.6 Orthogonal Procrustes Rotation......Page 214
13.7 Bibliographical Notes......Page 215
14.1 Block Matrices......Page 218
14.2 Canonical Correlation Analysis......Page 221
14.3 Generalized Canonical Correlation Analysis......Page 224
14.4 Multivariate Categorical Data......Page 228
14.5 Multiple Correspondence Analysis......Page 229
14.6 Homogeneity Assumption......Page 231
14.7 Bibliographical Notes......Page 233
15.1 Modification of Multiple Correspondence Analysis......Page 236
15.2 Canonical Discriminant Analysis......Page 238
15.3 Minimum Distance Classification......Page 240
15.4 Maximum Probability Classification......Page 241
15.5 Normal Discrimination for Two Groups......Page 243
15.6 Interpreting Solutions......Page 245
15.7 Generalized Normal Discrimination......Page 248
15.8 Bibliographical Notes......Page 251
16.1 Linking Coordinates to Quasi-distances......Page 253
16.2 Illustration of an MDS Solution......Page 255
16.3 Iterative Algorithm......Page 256
16.4 Matrix Expression for Squared Distances......Page 257
16.5 Inequality for Distances......Page 259
16.6 Majorization Algorithm......Page 261
16.7 Bibliographical Notes......Page 262
Advanced Procedures......Page 265
17.1 Introductory Systems of Linear Equations......Page 266
17.2 Moore–Penrose Inverse and System of Linear Equations......Page 267
17.3 Singular Value Decomposition and the Moore–Penrose Inverse......Page 269
17.4 Least Squares Problem Solved with Moore–Penrose Inverse......Page 271
17.5 Orthogonal Complement Matrix......Page 273
17.6 Kronecker Product......Page 276
17.7 Khatri–Rao Product......Page 277
17.8 Vec Operator......Page 279
17.9 Hadamard Product......Page 280
17.10 Bibliographical Notes......Page 281
18.1 Matrix Decomposition Formulation......Page 283
18.2 Comparisons to Latent Variable Formulation......Page 286
18.3 Solution of Loadings and Unique Variances......Page 287
18.4 Iterative Algorithm......Page 288
18.5 Estimation of Covariances Between Variables and Factor Scores......Page 289
18.6 Estimation of Loadings and Unique Variances......Page 292
18.7 Identifiability of the Model Part and Residuals......Page 293
18.8 Factor Scores as Higher Rank Approximations......Page 295
18.9 Bibliographical Notes......Page 297
19.1 Motivational Examples......Page 300
19.2 Comparisons of Models......Page 301
19.3 Solutions and Decomposition of the Sum of Squares......Page 303
19.4 Larger Common Part of Principal Component Analysis......Page 307
19.5 Better Fit of Factor Analysis......Page 308
19.6 Largeness of Unique Variances in Factor Analysis......Page 309
19.7 Inequalities for Latent Variable Factor Analysis......Page 310
19.8 Inequalities After Nonsingular Transformation......Page 311
19.9 Proofs for Inequalities......Page 312
19.10 Bibliographical Notes......Page 313
20.1 Tucker3 and Parafac Models......Page 314
20.2 Hierarchical Relationships Among PCA and 3WPCA......Page 316
20.3 Parafac Solution......Page 322
20.4 Tucker3 Solution......Page 325
20.5 Unconstrained Parafac Algorithm......Page 327
20.6 Constrained Parafac Algorithm......Page 330
20.7 Tucker3 Algorithm: The Optimal Core Array......Page 332
20.8 Tucker3 Algorithm: Iterative Solution......Page 334
20.9 Three-Way Rotation in Tucker3......Page 336
20.10 Bibliographical Notes......Page 339
21.1 Illustration of Sparse Solution......Page 343
21.2 Penalized Least Squares Method and Lasso......Page 345
21.3 Coordinate Descent Algorithm for Lasso......Page 346
21.4 Selection of Penalty Weight......Page 350
21.5 L0 Sparse Regression......Page 353
21.6 Standard Regression in Ordinary and High-Dimensional Cases......Page 355
21.7 High-Dimensional Variable Selection by Sparse Regression......Page 358
21.8 Bibliographical Notes......Page 360
22.1 From Confirmatory FA to Sparse FA......Page 362
22.2 Formulation of Penalized Sparse LVFA......Page 364
22.3 Algorithm for Penalized Sparse LVFA......Page 365
22.4 M-Step for Penalized Sparse LVFA......Page 366
22.5 Using Penalized Sparse LVFA......Page 370
22.6 Formulation of Cardinality Constrained MDFA......Page 373
22.7 Algorithm for Cardinality Constrained MDFA......Page 375
22.8 Using Cardinality Constrained MDFA......Page 376
22.9 Sparse FA Versus Factor Rotation in Exploratory FA......Page 378
22.10 Bibliographical Notes......Page 380
A.1.1 Angles Between Vectors......Page 384
Outline placeholder......Page 0
A.1.2 Orthonormal Matrix......Page 385
A.1.3 Vector Space......Page 386
A.1.4 Projection Onto a Subspace......Page 389
A.2.1 Decomposition Using Averages......Page 390
A.2.2 Decomposition Using a Projection Matrix......Page 392
A.3.1 SVD: Extended Version......Page 393
A.3.2 SVD: Compact Version......Page 395
A.3.3 Other Expressions of SVD......Page 396
A.3.4 SVD and Eigenvalue Decomposition for Sysmmetric Matrices......Page 398
A.4.1 ten Berge’s Theorem with Suborthonormal Matrices......Page 399
A.4.2 Maximization of Trace Functions......Page 400
A.4.3 Reduced Rank Approximation......Page 403
A.4.4 Modified Reduced Rank Approximation......Page 404
A.4.5 Modified Versions of Maximizing Trace Functions......Page 408
A.4.6 Obtaining Symmetric Square Roots of Matrices......Page 411
A.5.1 Estimates of Means and Covariances......Page 412
A.5.2 Multiple Groups with Homogeneous Covariances......Page 414
A.6.1 General Methodology......Page 416
A.6.2 Gradient Algorithm for Single Parameter Cases......Page 418
A.6.3 Gradient Algorithm for Multiple Parameter Cases......Page 420
A.7.1 Definition of Scale Invariance......Page 421
A.7.2 Scale Invariance of Factor Analysis......Page 423
A.7.3 Scale Invariance of Path Analysis......Page 425
A.8.1 Joint, Conditional, and Marginal Probability Densities......Page 426
A.8.2 Expected Values......Page 427
A.8.3 Covariances as Expected Values......Page 428
A.8.5 EM Algorithm......Page 430
A.9 EM Algorithm for Factor Analysis......Page 432
A.9.2 Complete Data Log Likelihood......Page 433
A.9.3 Expected Complete Data Log Likelihood......Page 435
A.9.4 E-Step......Page 436
A.9.5 Updating Unique Variances in M-Step......Page 438
A.9.6 Updating Factor Covariance in M-Step for CFA......Page 439
A.9.7 Updating Loadings in M-Step for EFA and CFA......Page 440
A.9.8 Whole Steps in EFA and CFA......Page 441
A.9.9 Algorithm for Penalized Factor Analysis......Page 442
References......Page 444
Index......Page 451