Feature Learning and Understanding: Algorithms and Applications

Preface
Contents
Notation
Chapter 1: A Gentle Introduction to Feature Learning
	1.1 Introduction
	1.2 Data and Preprocessing
		1.2.1 Data Collection
		1.2.2 Data Cleaning
		1.2.3 Data Sampling
		1.2.4 Data Transformation
	1.3 Feature Learning
		1.3.1 Solutions to Eigenvalue Equations
		1.3.2 Convex Optimization
		1.3.3 Gradient Descent
	1.4 Summary
Chapter 2: Latent Semantic Feature Extraction
	2.1 Introduction
	2.2 Singular Value Decomposition
		2.2.1 Feature Extraction by SVD
		2.2.2 An Example of SVD
	2.3 SVD Updating
	2.4 SVD with Compressive Sampling
	2.5 Case Studies
		2.5.1 Analysis of Coil-20 Data Set
		2.5.2 Latent Semantic Feature Extraction for Recommendation
	2.6 Summary
Chapter 3: Principal Component Analysis
	3.1 Introduction
	3.2 Classical Principal Component Analysis
		3.2.1 Maximizing Variance and Minimizing Residuals
		3.2.2 Theoretical Derivation of PCA
		3.2.3 An Alternative View of PCA
		3.2.4 Selection of the Reduced Dimension
		3.2.5 Eigendecomposition of XXT or XTX
		3.2.6 Relationship between PCA and SVD
	3.3 Probabilistic Principal Component Analysis
		3.3.1 Latent Variable Model
		3.3.2 The Probability Model of PPCA
		3.3.3 The Maximum Likelihood Estimation of PPCA
		3.3.4 The PPCA Algorithm
	3.4 Case Studies
		3.4.1 Enterprise Profit Ratio Analysis Using PCA
		3.4.2 Fault Detection Based on PCA
	3.5 Summary
Chapter 4: Manifold-Learning-Based Feature Extraction
	4.1 Introduction
	4.2 Manifold Learning and Spectral Graph Theory
	4.3 Neighborhood Preserving Projection
		4.3.1 Locally Linear Embedding (LLE)
		4.3.2 Neighborhood Preserving Embedding (NPE)
	4.4 Locality Preserving Projection (LPP)
		4.4.1 Relationship to PCA
		4.4.2 Relationship to Laplacian Eigenmaps
	4.5 Case Studies
		4.5.1 Handwritten Digit Visualization
		4.5.2 Face Manifold Analysis
	4.6 Summary
Chapter 5: Linear Discriminant Analysis
	5.1 Introduction
	5.2 Fisher´s Linear Discriminant
	5.3 Analysis of FLD
	5.4 Linear Discriminant Analysis
		5.4.1 An Example of LDA
		5.4.2 Foley-Sammon Optimal Discriminant Vectors
	5.5 Case Study
	5.6 Summary
Chapter 6: Kernel-Based Nonlinear Feature Learning
	6.1 Introduction
	6.2 Kernel Trick
	6.3 Kernel Principal Component Analysis
		6.3.1 Revisiting of PCA
		6.3.2 Derivation of Kernel Principal Component Analysis
		6.3.3 Kernel Averaging Filter
	6.4 Kernel Fisher Discriminant
	6.5 Generalized Discriminant Analysis
	6.6 Case Study
	6.7 Summary
Chapter 7: Sparse Feature Learning
	7.1 Introduction
	7.2 Sparse Representation Problem with Different Norm Regularizations
		7.2.1 0-norm Regularized Sparse Representation
		7.2.2 1-norm Regularized Sparse Representation
		7.2.3 p-norm (0 < p < 1) Regularized Sparse Representation
		7.2.4 2,1-norm Regularized Group-Wise Sparse Representation
	7.3 Lasso Estimator
	7.4 Sparse Feature Learning with Generalized Regression
		7.4.1 Sparse Principal Component Analysis
		7.4.2 Generalized Robust Regression (GRR) for Jointly Sparse Subspace Learning
		7.4.3 Robust Jointly Sparse Regression with Generalized Orthogonal Learning for Image Feature Selection
		7.4.4 Locally Joint Sparse Marginal Embedding for Feature Extraction
	7.5 Case Study
	7.6 Summary
Chapter 8: Low Rank Feature Learning
	8.1 Introduction
	8.2 Low Rank Approximation Problems
	8.3 Low Rank Projection Learning Algorithms
	8.4 Robust Low Rank Projection Learning
		8.4.1 Low-Rank Preserving Projections
		8.4.2 Low-Rank Preserving Projection with GRR
		8.4.3 Low-Rank Linear Embedding
		8.4.4 Feature Selective Projection with Low-Rank Embedding and Dual Laplacian Regularization
	8.5 Case Study
		8.5.1 Databases
		8.5.2 Observations and Discussions
	8.6 Summary
Chapter 9: Tensor-Based Feature Learning
	9.1 Introduction
	9.2 Tensor Representation Based on Tucker Decomposition
		9.2.1 Preliminaries of Tucker Decomposition
		9.2.2 Main Idea of Tucker-Based Feature Learning
	9.3 Rationality: Criteria for Tucker-Based Feature Learning Models
		9.3.1 Least Square Error Multi-linear Representation: Tucker-Based PCA
		9.3.2 Living in a Manifold: Tucker-Based Manifold Learning
		9.3.3 Learning with the Truth: Tucker-Based Discriminant Analysis
	9.4 Solvability: An Algorithmic Framework of Alternative Minimization
		9.4.1 Alternative Minimization Algorithms
		9.4.2 A Unified Framework
		9.4.3 Sparsity Helps: Sparse Tensor Alignment
	9.5 Case Study
		9.5.1 Alternative Minimization for MJSPCA
		9.5.2 Action Recognition with MJSPCA
	9.6 Summary
Chapter 10: Neural-Network-Based Feature Learning: Auto-Encoder
	10.1 Introduction
	10.2 Auto-Encoder (AE)
		10.2.1 Fully Connected Layer and Activation Function
		10.2.2 Basic Auto-Encoder
		10.2.3 Backpropagation and Computational Graphs
		10.2.4 Relationship Between the Dimension of Data and the Dimension of Feautures
	10.3 Denoising Auto-Encoder (DAE)
	10.4 Stacked Auto-Encoder
		10.4.1 Training Stacked Auto-Encoder
		10.4.2 Stacked Denoising Auto-Encoders (SDAE)
	10.5 Applications of Auto-Encoders
	10.6 Case Studies
		10.6.1 Auto-Encoder for Feature Learning
		10.6.2 Auto-Encoder for Fault Detection
	10.7 Summary
Chapter 11: Neural-Network-Based Feature Learning: Convolutional Neural Network
	11.1 Introduction
	11.2 Basic Architecture of CNNs
		11.2.1 Convolutional Layer
		11.2.2 Pooling Layer
		11.2.3 Batch Normalization
		11.2.4 Dropout
		11.2.5 Relationship between Convolutional Layer and Fully Connected Layer
		11.2.6 Backpropagation of Convolutional Layers
	11.3 Transfer Feature Learning of CNN
		11.3.1 Formalization of Transfer Learning Problems
		11.3.2 Basic Method of Transfer Learning
	11.4 Deep Convolutional Models
		11.4.1 The Beginning of Deep Convolutional Neural Networks: AlexNet
		11.4.2 Common Architecture: VGG
		11.4.3 Inception Mechanism: GoogLeNet
		11.4.4 Stacked Convolutional Auto-Encoders
	11.5 Case Studies
		11.5.1 CNN-Based Handwritten Numeral Recognition
		11.5.2 Spatial Transformer Network
	11.6 Summary
Chapter 12: Neural-Network-Based Feature Learning: Recurrent Neural Network
	12.1 Introduction
	12.2 Recurrent Neural Networks
		12.2.1 Forward Propagation
		12.2.2 Backpropagation Through Time (BPTT)
		12.2.3 Different Types of RNNs
	12.3 Long Short-Term Memory (LSTM)
		12.3.1 Forget Gate
		12.3.2 Input Gate
		12.3.3 Output Gate
		12.3.4 The Backpropagation of LSTM
		12.3.5 Explanation of Gradient Vanishing
	12.4 Gated Recurrent Unit (GRU)
	12.5 Deep RNNs
	12.6 Case Study
		12.6.1 Datasets Introduction
		12.6.2 Data Preprocessing
		12.6.3 Define Network Architecture and Training Options
		12.6.4 Test the Networks
	12.7 Summary
References
Index