Practical Guide to Chemometrics

PRACTICAL GUIDE to CHEMOMETRICS, SECOND EDITION......Page 1
Preface......Page 3
Editor......Page 4
Contributors......Page 5
Contents......Page 6
Table of Contents......Page 0
1.1 CHEMICAL MEASUREMENTS — A BASIS FOR DECISION MAKING......Page 8
1.3 CHEMOMETRICS......Page 9
1.4 HOW TO USE THIS BOOK......Page 10
1.4.1 SOFTWARE APPLICATIONS......Page 11
BOOKS......Page 12
REFERENCES......Page 13
CONTENTS......Page 14
INTRODUCTION......Page 15
2.1 SOURCES OF ERROR......Page 16
2.1.1 SOME COMMON TERMS......Page 17
2.2 PRECISION AND ACCURACY......Page 19
2.3 PROPERTIES OF THE NORMAL DISTRIBUTION......Page 21
2.4 SIGNIFICANCE TESTING......Page 25
2.4.1 THE F-TEST FOR COMPARISON OF VARIANCE (PRECISION)......Page 26
2.4.2 THE STUDENT T-TEST......Page 29
2.4.4 COMPARISON OF A SAMPLE MEAN WITH A CERTIFIED VALUE......Page 31
2.4.5 COMPARISON OF THE MEANS FROM TWO SAMPLES......Page 32
2.4.6 COMPARISON OF TWO METHODS WITH DIFFERENT TEST OBJECTS OR SPECIMENS......Page 33
2.5 ANALYSIS OF VARIANCE......Page 34
2.5.1 ANOVA TO TEST FOR DIFFERENCES BETWEEN MEANS......Page 35
2.5.3 BETWEEN-SAMPLE VARIATION (BETWEEN-TREATMENT VARIATION)......Page 36
2.5.4 ANALYSIS OF RESIDUALS......Page 37
2.6 OUTLIERS......Page 40
2.7 ROBUST ESTIMATES OF CENTRAL TENDENCY AND SPREAD......Page 43
2.8 SOFTWARE......Page 45
2.8.1 ANOVA USING EXCEL......Page 46
REFERENCES......Page 47
CONTENTS......Page 48
3.1 SAMPLING AND SAMPLING DISTRIBUTIONS......Page 49
3.1.1 THE NORMAL DISTRIBUTION......Page 50
3.2.1 IMPLICATIONS OF THE CENTRAL LIMIT THEOREM......Page 52
3.3.1 THE T-DISTRIBUTION......Page 53
3.3.2 CHI-SQUARE DISTRIBUTION......Page 54
3.4 UNIVARIATE HYPOTHESIS TESTING......Page 55
3.4.1 INFERENCES ABOUT MEANS......Page 56
3.5 THE MULTIVARIATE NORMAL DISTRIBUTION......Page 58
3.5.1 GENERALIZED OR MAHALANOBIS DISTANCES......Page 59
3.5.2 THE VARIANCE–COVARIANCE MATRIX......Page 60
3.5.3 ESTIMATION OF POPULATION PARAMETERS FROM SMALL SAMPLES......Page 61
3.5.5 GENERALIZED SAMPLE VARIANCE......Page 62
3.5.6 GRAPHICAL ILLUSTRATION OF SELECTED BIVARIATE NORMAL DISTRIBUTIONS......Page 63
3.5.7 CHI-SQUARE DISTRIBUTION......Page 65
3.7 EXAMPLE: MULTIVARIATE DISTANCES......Page 66
3.7.1 STEP 1: GRAPHICAL REVIEW OF SMX.MAT DATA FILE......Page 67
3.7.3 STEP 3: VIEW HISTOGRAMS OF SELECTED VARIABLES......Page 68
3.7.4 STEP 4: COMPUTE THE TRAINING SET MEAN AND VARIANCE–COVARIANCE MATRIX......Page 69
3.7.5 STEP 5: CALCULATE MAHALANOBIS DISTANCES AND PROBABILITY DENSITIES......Page 71
3.7.6 STEP 6: FIND “ACCEPTABLE” AND “UNACCEPTABLE” OBJECTS......Page 72
RECOMMENDED READING......Page 73
REFERENCES......Page 74
CONTENTS......Page 75
4.2 SPECTROSCOPIC-CHROMATOGRAPHIC DATA......Page 76
4.2.1 BASIS VECTORS......Page 77
4.3 THE PRINCIPAL COMPONENT MODEL......Page 79
4.3.1 EIGENVECTORS AND EIGENVALUES......Page 80
4.3.2 THE SINGULAR-VALUE DECOMPOSITION......Page 82
4.4 PREPROCESSING OPTIONS......Page 83
4.4.2 VARIANCE SCALING......Page 84
4.4.3 BASELINE CORRECTION......Page 86
4.4.4 SMOOTHING AND FILTERING......Page 87
4.4.5 FIRST AND SECOND DERIVATIVES......Page 88
4.4.7 MULTIPLICATIVE SCATTER CORRECTION (MSC) AND STANDARD NORMAL VARIATE (SNV) TRANSFORMS......Page 89
4.5 PCA DATA EXPLORATION PROCEDURE......Page 92
4.6 INFLUENCING FACTORS......Page 93
4.6.1 VARIANCE AND RESIDUAL VARIANCE......Page 95
4.6.3 F-TEST FOR DETERMINING THE NUMBER OF FACTORS......Page 99
4.7 BASIS VECTORS......Page 102
4.8 RESIDUAL SPECTRA......Page 104
4.8.1 RESIDUAL VARIANCE ANALYSIS......Page 106
4.9 CONCLUSIONS......Page 108
REFERENCES......Page 109
CONTENTS......Page 111
5.1.1 NEAR INFRARED SPECTROSCOPY......Page 113
5.1.4 SOLVENT INTERACTIONS......Page 114
5.2.1 UNIVARIATE CALIBRATION......Page 115
5.2.2 NONZERO INTERCEPTS......Page 116
5.2.3 MULTIVARIATE CALIBRATION......Page 117
5.2.4 CURVILINEAR CALIBRATION......Page 118
5.2.5 SELECTION OF CALIBRATION AND VALIDATION SAMPLES......Page 119
5.2.6 MEASUREMENT ERROR AND MEASURES OF PREDICTION ERROR......Page 120
5.3.1 GRAPHICAL SURVEY OF NIR WATER–METHANOL DATA......Page 122
5.3.2.1 Without an Intercept Term......Page 124
5.3.3 MULTIVARIATE CALIBRATION......Page 125
5.4 STATISTICAL EVALUATION OF CALIBRATION MODELS OBTAINED BY LEAST SQUARES......Page 127
5.4.1 HYPOTHESIS TESTING......Page 128
5.4.2 PARTITIONING OF VARIANCE IN LEAST-SQUARES SOLUTIONS......Page 129
5.4.3 INTERPRETING REGRESSION ANOVA TABLES......Page 131
5.4.4 CONFIDENCE INTERVAL AND HYPOTHESIS TESTS FOR REGRESSION COEFFICIENTS......Page 132
5.4.5 PREDICTION CONFIDENCE INTERVALS......Page 133
5.4.6 LEVERAGE AND INFLUENCE......Page 134
5.4.7 MODEL DEPARTURES AND OUTLIERS......Page 135
5.4.8 COEFFICIENT OF DETERMINATION AND MULTIPLE CORRELATION COEFFICIENT......Page 136
5.4.9.1 Sensitivity......Page 137
5.4.9.2.1 Univariate Decision Limit......Page 138
5.4.9.2.2 Univariate Detection Limit......Page 139
5.4.10 INTERFERENCE EFFECTS AND SELECTIVITY......Page 140
5.5 VARIABLE SELECTION......Page 141
5.5.2.1 Variable-Addition Step......Page 142
5.5.3 BACKWARD ELIMINATION......Page 143
5.5.7 RECOMMENDATIONS AND PRECAUTIONS......Page 144
5.6 BIASED METHODS OF CALIBRATION......Page 145
5.6.1 PRINCIPAL COMPONENT REGRESSION......Page 146
5.6.1.1 Basis Vectors......Page 147
5.6.1.2.1 Calibration Steps......Page 148
5.6.1.2.2 Unknown Prediction Steps......Page 149
5.6.1.3 Number of Basis Vectors......Page 150
5.6.1.4 Example PCR Results......Page 151
5.6.2 PARTIAL LEAST SQUARES......Page 153
5.6.2.1 Mathematical Procedure......Page 154
5.6.2.3 Comparison with PCR......Page 155
5.6.3.1 Common Basis Vectors and a Generic Model......Page 156
5.6.4 REGULARIZATION......Page 157
5.7 STANDARD ADDITION METHOD......Page 159
5.7.1 UNIVARIATE STANDARD ADDITION METHOD......Page 160
5.7.2 MULTIVARIATE STANDARD ADDITION METHOD......Page 161
5.9 PREPROCESSING TECHNIQUES......Page 162
5.10.1 STANDARDIZATION OF PREDICTED VALUES......Page 163
5.10.2 STANDARDIZATION OF INSTRUMENT RESPONSE......Page 164
5.11 SOFTWARE......Page 165
REFERENCES......Page 166
CONTENTS......Page 172
6.1 INTRODUCTION......Page 173
6.2.1 THE MEAN AND THE STANDARD DEVIATION......Page 174
6.2.3 OTHER ROBUST ESTIMATORS OF LOCATION AND SCALE......Page 176
6.3.1 THE EMPIRICAL MEAN AND COVARIANCE MATRIX......Page 178
6.3.2 THE ROBUST MCD ESTIMATOR......Page 179
6.4.1.1 The Multiple Linear Regression Model......Page 181
6.4.1.2 The Classical Least-Squares Estimator......Page 182
6.4.1.3 The Robust LTS Estimator......Page 183
6.4.1.4 An Outlier Map......Page 185
6.4.1.5 Other Robust Regression Estimators......Page 187
6.4.2.1 The Multivariate Linear Regression Model......Page 188
6.4.2.2 The Robust MCD-Regression Estimator......Page 189
6.5.1 CLASSICAL PCA......Page 190
6.5.2 ROBUST PCA BASED ON A ROBUST COVARIANCE ESTIMATOR......Page 192
6.5.3 ROBUST PCA BASED ON PROJECTION PURSUIT......Page 193
6.5.4 ROBUST PCA BASED ON PROJECTION PURSUIT AND THE MCD......Page 194
6.5.5 AN OUTLIER MAP......Page 196
6.5.6 SELECTING THE NUMBER OF PRINCIPAL COMPONENTS......Page 198
6.6.1 CLASSICAL PCR......Page 199
6.6.2 ROBUST PCR......Page 202
6.6.3 MODEL CALIBRATION AND VALIDATION......Page 203
6.6.4 AN EXAMPLE......Page 204
6.7.1 CLASSICAL PLSR......Page 207
6.7.2 ROBUST PLSR......Page 208
6.7.3 AN EXAMPLE......Page 209
6.8.1.1 Classical and Robust Discriminant Rules......Page 212
6.8.1.2 Evaluating the Discriminant Rules......Page 213
6.8.1.3 An Example......Page 214
6.9 SOFTWARE AVAILABILITY......Page 216
REFERENCES......Page 217
CONTENTS......Page 221
7.1 INTRODUCTION......Page 222
7.2 MULTIVARIATE DATA, BEER-LAMBERT’S LAW, MATRIX NOTATION......Page 223
7.3 CALCULATION OF THE CONCENTRATION PROFILES: CASE I, SIMPLE MECHANISMS......Page 224
7.4 MODEL-BASED NONLINEAR FITTING......Page 226
7.4.1 DIRECT METHODS, SIMPLEX......Page 229
7.4.2 NONLINEAR FITTING USING EXCEL’S SOLVER......Page 231
7.4.3 LINEAR AND NONLINEAR PARAMETERS......Page 232
7.4.4 NEWTON-GAUSS-LEVENBERG/MARQUARDT (NGL/M)......Page 234
7.4.5 NONWHITE NOISE......Page 241
7.5 CALCULATION OF THE CONCENTRATION PROFILES: CASE II, COMPLEX MECHANISMS......Page 245
7.5.1 FOURTH-ORDER RUNGE-KUTTA METHOD IN EXCEL......Page 246
7.5.2.1 Autocatalysis......Page 250
7.5.2.2 Zeroth-Order Reaction......Page 252
7.5.2.3 Lotka-Volterra (Sheep and Wolves)......Page 254
7.5.2.4 The Belousov-Zhabotinsky (BZ) Reaction......Page 255
7.6 CALCULATION OF THE CONCENTRATION PROFILES: CASE III, VERY COMPLEX MECHANISMS......Page 257
7.7 RELATED ISSUES......Page 259
7.7.4 GLOBALIZATION OF THE ANALYSIS......Page 260
7.7.5 SOFT-MODELING METHODS......Page 261
APPENDIX......Page 262
REFERENCES......Page 263
CONTENTS......Page 266
8.1 INTRODUCTION......Page 267
8.2.1 THE GENERAL SCHEME OF RSM......Page 268
8.2.2.1 Process Factor Spaces......Page 271
8.2.2.2 Mixture Factor Spaces......Page 272
8.2.2.3 Simplex-Lattice Designs......Page 275
8.2.2.3.2 Disadvantages of the Simplex-Lattice Designs......Page 277
8.2.2.4 Simplex-Centroid Designs......Page 278
8.2.2.4.2 Disadvantages of Simplex-Centroid Designs......Page 280
8.2.2.4.3 Simplex-Centroid Design, Example......Page 281
8.2.2.5 Constrained Mixture Spaces......Page 282
8.2.2.6 Mixture+Process Factor Spaces......Page 286
8.2.3 SOME REGRESSION-ANALYSIS-RELATED NOTATION......Page 289
8.3.1 BIVARIATE (MULTIVARIATE) EXAMPLE......Page 291
8.4.1.1 Advantages of Factorial Designs......Page 293
8.4.2 THREE OR MORE LEVELS IN FULL FACTORIAL DESIGNS......Page 294
8.4.3 CENTRAL COMPOSITE DESIGNS......Page 296
8.5 THE TAGUCHI EXPERIMENTAL DESIGN APPROACH......Page 297
8.6.1 OPTIMALITY CRITERIA......Page 301
8.6.2 OPTIMAL VS. EQUALLY DISTANCED DESIGNS......Page 302
8.6.3 DESIGN OPTIMALITY AND DESIGN EFFICIENCY CRITERIA......Page 305
8.6.3.1 Design Measures......Page 306
8.6.3.2 D-Optimality and D-Efficiency......Page 307
8.6.3.3 G-Optimality and G-Efficiency......Page 308
8.7 ALGORITHMS FOR THE SEARCH OF REALIZABLE OPTIMAL EXPERIMENTAL DESIGNS......Page 309
8.7.1.1 Fedorov’s Algorithm......Page 310
8.7.1.3 DETMAX Algorithm......Page 311
8.7.1.4 The MD Galil and Kiefer’s Algorithm......Page 312
8.7.2 SEQUENTIAL D-OPTIMAL DESIGNS......Page 313
8.7.2.1 Example......Page 314
8.7.3 SEQUENTIAL COMPOSITE D-OPTIMAL DESIGNS......Page 316
8.8.1.1 MATLAB......Page 319
8.8.1.3 Other Packages......Page 322
8.8.2 CATALOGS OF EXPERIMENTAL DESIGNS......Page 323
8.9.1 CONSTRUCTION OF A CALIBRATION SAMPLE SET......Page 324
8.9.1.1 Identifying of the Number of Significant Factors......Page 325
8.9.1.2 Identifying the Type of the Regression Model......Page 328
8.9.1.3 Defining the Bounds of the Factor Space......Page 330
8.9.1.4 Estimating Extinction Coefficients......Page 332
8.9.2 IMPROVING QUALITY FROM HISTORICAL DATA......Page 333
8.9.2.1 Improving the Numerical Stability of the Data Set......Page 336
8.9.2.2 Prediction Ability......Page 337
REFERENCES......Page 340
9.1 INTRODUCTION......Page 342
9.2 DATA PREPROCESSING......Page 344
9.3 MAPPING AND DISPLAY......Page 345
9.4 CLUSTERING......Page 350
9.5 CLASSIFICATION......Page 354
9.5.2 PARTIAL LEAST SQUARES......Page 355
9.5.3 SIMCA......Page 356
9.6 PRACTICAL CONSIDERATIONS......Page 357
9.7 APPLICATIONS OF PATTERN-RECOGNITION TECHNIQUES......Page 358
9.7.1 ARCHAEOLOGICAL ARTIFACTS......Page 359
9.7.2 FUEL SPILL IDENTIFICATION......Page 361
9.7.3 SORTING PLASTICS FOR RECYCLING......Page 368
9.7.4 TAXONOMY BASED ON CHEMICAL CONSTITUTION......Page 374
REFERENCES......Page 377
10.1 INTRODUCTION......Page 381
10.2 NOISE REMOVAL AND THE PROBLEM OF PRIOR INFORMATION......Page 382
10.2.1 SIGNAL ESTIMATION AND SIGNAL DETECTION......Page 383
10.3 REEXPRESSING DATA IN ALTERNATE BASES TO ANALYZE STRUCTURE......Page 384
10.3.1 PROJECTION-BASED SIGNAL ANALYSIS AS SIGNAL PROCESSING......Page 385
10.4 FREQUENCY-DOMAIN SIGNAL PROCESSING......Page 387
10.4.2 THE SAMPLING THEOREM AND ALIASING......Page 388
10.4.3 THE BANDWIDTH-LIMITED, DISCRETE FOURIER TRANSFORM......Page 390
10.4.4 PROPERTIES OF THE FOURIER TRANSFORM......Page 391
10.5.1 SMOOTHING......Page 396
10.5.2 SMOOTHING WITH DESIGNER TRANSFER FUNCTIONS......Page 397
10.6.1 SMOOTHING......Page 400
10.6.2 FILTERING......Page 402
10.6.3 POLYNOMIAL MOVING-AVERAGE (SAVITSKY-GOLAY) FILTERS......Page 405
10.7.1 THE WAVELET FUNCTION......Page 408
10.7.2 TIME AND FREQUENCY LOCALIZATIONS OF WAVELET FUNCTIONS......Page 410
10.7.3 THE DISCRETE WAVELET TRANSFORM......Page 411
10.7.4 SMOOTHING AND DENOISING WITH WAVELETS......Page 414
REFERENCES......Page 416
FURTHER READING......Page 418
CONTENTS......Page 419
11.1 INTRODUCTION: GENERAL CONCEPT, AMBIGUITIES, RESOLUTION THEOREMS......Page 420
11.2 HISTORICAL BACKGROUND......Page 424
11.3 LOCAL RANK AND RESOLUTION: EVOLVING FACTOR ANALYSIS AND RELATED TECHNIQUES......Page 425
11.4 NONITERATIVE RESOLUTION METHODS......Page 428
11.4.1 WINDOW FACTOR ANALYSIS (WFA)......Page 429
11.4.2 OTHER TECHNIQUES: SUBWINDOW FACTOR ANALYSIS (SFA) AND HEURISTIC EVOLVING LATENT PROJECTIONS (HELP)......Page 431
11.5 ITERATIVE METHODS......Page 433
11.5.1 GENERATION OF INITIAL ESTIMATES......Page 434
11.5.2 CONSTRAINTS, DEFINITION, CLASSIFICATION: EQUALITY AND INEQUALITY CONSTRAINTS BASED ON CHEMICAL OR MATHEMATICAL PROPERTIES......Page 435
11.5.2.3 Closure......Page 436
11.5.2.6 Local-Rank Constraints, Selectivity, and Zero-Concentration Windows......Page 437
11.5.3 ITERATIVE TARGET TRANSFORMATION FACTOR ANALYSIS (ITTFA)......Page 439
11.5.4 MULTIVARIATE CURVE RESOLUTION-ALTERNATING LEAST SQUARES (MCR-ALS)......Page 441
11.6 EXTENSION OF SELF-MODELING CURVE RESOLUTION TO MULTIWAY DATA: MCR-ALS SIMULTANEOUS ANALYSIS OF MULTIPLE CORRELATED DATA MATRICES......Page 442
11.7 UNCERTAINTY IN RESOLUTION RESULTS, RANGE OF FEASIBLE SOLUTIONS, AND ERROR IN RESOLUTION......Page 448
11.8 APPLICATIONS......Page 450
11.8.1 BIOCHEMICAL PROCESSES......Page 451
11.8.1.1 Study of Changes in the Protein Secondary Structure......Page 453
11.8.1.3 Global Description of the Protein Folding Process......Page 455
11.8.2 ENVIRONMENTAL DATA......Page 456
11.8.3 SPECTROSCOPIC IMAGES......Page 463
11.9 SOFTWARE......Page 467
REFERENCES......Page 469
12.1 INTRODUCTION......Page 476
12.2 BACKGROUND......Page 477
12.4 THREE-WAY MODELS......Page 479
12.5 EXAMPLES......Page 482
12.6.1 RANK ANNIHILATION FACTOR ANALYSIS......Page 483
12.6.1.1 RAFA Application......Page 484
12.6.2 GENERALIZED RANK ANNIHILATION METHOD......Page 486
12.6.2.1 GRAM Application......Page 487
12.6.3 DIRECT TRILINEAR DECOMPOSITION......Page 490
12.6.3.1 DTLD Application......Page 491
12.7.1 PARAFAC / CANDECOMP......Page 492
12.7.1.2 Solution Constraints......Page 494
12.7.1.3 PARAFAC Application......Page 495
12.8 EXTENSIONS OF THREE-WAY METHODS......Page 496
12.9 FIGURES OF MERIT......Page 497
12.10 CAVEATS......Page 498
REFERENCES......Page 500
APPENDIX 12.1 GRAM ALGORITHM......Page 503
APPENDIX 12.2 DTLD ALGORITHM......Page 504
APPENDIX 12.3 PARAFAC ALGORITHM......Page 505
CONTENTS......Page 509
13.1 HISTORICAL DEVELOPMENT OF CHEMOMETRICS......Page 510
13.2 REVIEWS OF CHEMOMETRICS AND FUTURE TRENDS......Page 511
13.2.3 FOOD AND FEED CHEMISTRY......Page 512
13.3 DRIVERS OF GROWTH IN CHEMOMETRICS......Page 513
13.3.2 CHEMOMETRICS AT THE INTERFACE OF CHEMICAL AND BIOLOGICAL SCIENCES......Page 514
REFERENCES......Page 516