Fundamentals of Pattern Recognition and Machine Learning

Preface to the Second Edition
Preface to the First Edition
Contents
Chapter 1 Introduction
	1.1 Pattern Recognition and Machine Learning
	1.2 Basic Mathematical Setting
	1.3 Prediction
	1.4 Prediction Error
	1.5 Supervised vs. Unsupervised Learning
	1.6 Complexity Trade-Offs
	1.7 The Design Cycle
	1.8 Application Examples
		1.8.1 Bioinformatics
		1.8.2 Materials Informatics
	1.9 Bibliographical Notes
Chapter 2 Optimal Classification
	2.1 Classification without Features
	2.2 Classification with Features
	2.3 The Bayes Classifier
	2.4 The Bayes Error
	2.5 Gaussian Model
		2.5.1 Homoskedastic Case
		2.5.2 Heteroskedastic Case
	2.6 Additional Topics
		2.6.1 Minimax Classification
		2.6.2 F-errors
		2.6.3 Bayes Decision Theory
		2.6.4 Rigorous Formulation of the Classification Problem
	2.7 Bibliographical Notes
	2.8 Exercises
	2.9 Python Assignments
Chapter 3 Sample-Based Classification
	3.1 Classification Rules
	3.2 Classification Error Rates
	3.3 Consistency
	3.4 No-Free-Lunch Theorems
	3.5 Additional Topics
		3.5.1 Ensemble Classification
		3.5.2 Mixture Sampling vs. Separate Sampling
	3.6 Bibliographical Notes
	3.7 Exercises
	3.8 Python Assignments
Chapter 4 Parametric Classification
	4.1 Parametric Plug-in Rules
	4.2 Gaussian Discriminant Analysis
		4.2.1 Linear Discriminant Analysis
		4.2.2 Quadratic Discriminant Analysis
	4.3 Logistic Classification
	4.4 Additional Topics
		4.4.1 Regularized Discriminant Analysis
		4.4.2 Consistency of Parametric Rules
		4.4.3 Bayesian Parametric Rules
	4.5 Bibliographical Notes
	4.6 Exercises
	4.7 Python Assignments
Chapter 5 Nonparametric Classification
	5.1 Nonparametric Plug-in Rules
	5.2 Histogram Classification
	5.3 Nearest-Neighbor Classification
	5.4 Kernel Classification
	5.5 Cover-Hart Theorem
	5.6 Stone’s Theorem
	5.7 Bibliographical Notes
	5.8 Exercises
	5.9 Python Assignments
Chapter 6 Function-Approximation Classification
	6.1 Support Vector Machines
		6.1.1 Linear SVMs for Separable Data
		6.1.2 General Linear SVMs
		6.1.3 Nonlinear SVMs
	6.2 Neural Networks
		6.2.1 Fully-Connected Neural Networks
		6.2.2 Neural Network Classifiers
		6.2.3 Loss Functions
		6.2.4 Backpropagation Algorithm
		6.2.5 Training Deep Neural Networks
		6.2.6 Convolutional Neural Networks
		6.2.7 Universal Approximation Property of Neural Networks
		6.2.8 Universal Consistency Theorems
	6.3 Decision Trees
	6.4 Rank-Based Classifiers
	6.5 Bibliographical Notes
	6.6 Exercises
	6.7 Python Assignments
Chapter 7 Error Estimation for Classification
	7.1 Error Estimation Rules
	7.2 Error Estimation Performance
		7.2.1 Deviation Distribution
		7.2.2 Bias, Variance, RMS, and Tail Probabilities
		7.2.3 Consistency
	7.3 Test-Set Error Estimation
	7.4 Resubstitution
	7.5 Cross-Validation
	7.6 Bootstrap
	7.7 Bolstered Error Estimation
	7.8 Additional Topics
		7.8.1 Convex Error Estimators
		7.8.2 Smoothed Error Estimators
		7.8.3 Bayesian Error Estimation
	7.9 Bibliographical Notes
	7.10 Exercises
	7.11 Python Assignments
Chapter 8 Model Selection for Classification
	8.1 Classification Complexity
	8.2 Vapnik-Chervonenkis Theory
		*8.2.1 Finite Model Selection
		8.2.2 Shatter Coefficients and VC Dimension
		8.2.3 VC Parameters of a Few Classification Rules
		8.2.4 Vapnik-Chervonenkis Theorem
		8.2.5 No-Free-Lunch Theorems
	8.3 Model Selection Methods
		8.3.1 Validation Error Minimization
		8.3.2 Training Error Minimization
		8.3.3 Structural Risk Minimization
	8.4 Bibliographical Notes
	8.5 Exercises
Chapter 9 Dimensionality Reduction
	9.1 Feature Extraction for Classification
	9.2 Feature Selection
		9.2.1 Exhaustive Search
		9.2.2 Univariate Greedy Search
		9.2.3 Multivariate Greedy Search
		9.2.4 Feature Selection and Classification Complexity
		9.2.5 Feature Selection and Error Estimation
	9.3 Principal Component Analysis (PCA)
	9.4 Multidimensional Scaling (MDS)
	9.5 Factor Analysis
	9.6 Bibliographical Notes
	9.7 Exercises
	9.8 Python Assignments
Chapter 10 Clustering
	10.1 K-Means Algorithm
	10.2 Gaussian Mixture Modeling
		10.2.1 Expectation-Maximization Approach
		10.2.2 Relationship to
	10.3 Hierarchical Clustering
	10.4 Self-Organizing Maps (SOM)
	10.5 Bibliographical Notes
	10.6 Exercises
	10.7 Python Assignments
Chapter 11 Regression
	11.1 Optimal Regression
	11.2 Sample-Based Regression
	11.3 Parametric Regression
		11.3.1 Linear Regression
		11.3.2 Gauss-Markov Theorem
		11.3.3 Penalized Least Squares
	11.4 Nonparametric Regression
		11.4.1 Kernel Regression
		11.4.2 Gaussian Process Regression
	11.5 Function-Approximation Regression
	11.6 Error Estimation
	11.7 Variable Selection
		11.7.1 Wrapper Search
		11.7.2 Statistical Testing
		11.7.3 LASSO and Elastic Net
	11.8 Model Selection
	11.9 Bibliographical Notes
	11.10 Exercises
	11.11 Python Assignments
Chapter 12 Physics-Informed Machine Learning
	12.1 The Relationship between Data and Theory
	12.2 Partial Differential Equation Models
	12.3 Physics-Informed Neural Networks
		12.3.1 Forward Problem
		12.3.2 Inverse Problem
		12.3.3 Hard Constraints
	12.4 Physics-Informed Gaussian Processes
		12.4.1 Forward Problem
		12.4.2 Inverse Problem
		12.4.3 Hard Constraints
	12.5 Bibliographical Notes
	12.6 Exercises
	12.7 Python Assignments
Appendix
	A1 Probability Theory
		A1.1 Sample Space and Events
		A1.2 Probability Measure
		A1.3 Conditional Probability and Independence
		A1.4 Random Variables
		A1.5 Joint and Conditional Distributions
		A1.6 Expectation
		A1.7 Vector Random Variables
		A1.8 Convergence of Random Sequences
		A1.9 Asymptotic Theorems
	A2 Basic Matrix Theory
	A3 Basic Lagrange-Multiplier Optimization
	A4 Proof of the Cover-Hart Theorem
	A5 Proof of Stone’s Theorem
	A6 Proof of the Vapnik-Chervonenkis Theorem
	A7 Proof of Convergence of the EM Algorithm
	A8 Data Sets Used in the Book
		A8.1 Synthetic Data
		A8.2 Dengue Fever Prognosis Data Set
		A8.3 Breast Cancer Prognosis Data Set
		A8.4 Stacking Fault Energy Data Set
		A8.5 Soft Magnetic Alloy Data Set
		A8.6 Ultrahigh Carbon Steel Data Set
List of Symbols
Bibliography
Index