Geometry of Deep Learning: A Signal Processing Perspective

Preface
Contents
Part I Basic Tools for Machine Learning
	1 Mathematical Preliminaries
		1.1 Metric Space
		1.2 Vector Space
		1.3 Banach and Hilbert Space
			1.3.1 Basis and Frames
		1.4 Probability Space
		1.5 Some Matrix Algebra
			1.5.1 Kronecker Product
			1.5.2 Matrix and Vector Calculus
		1.6 Elements of Convex Optimization
			1.6.1 Some Definitions
			1.6.2 Convex Sets, Convex Functions
			1.6.3 Subdifferentials
			1.6.4 Convex Conjugate
			1.6.5 Lagrangian Dual Formulation
		1.7 Exercises
	2 Linear and Kernel Classifiers
		2.1 Introduction
		2.2 Hard-Margin Linear Classifier
			2.2.1 Maximum Margin Classifier for Separable Cases
			2.2.2 Dual Formulation
			2.2.3 KKT Conditions and Support Vectors
		2.3 Soft-Margin Linear Classifiers
			2.3.1 Maximum Margin Classifier with Noise
		2.4 Nonlinear Classifier Using Kernel SVM
			2.4.1 Linear Classifier in the Feature Space
			2.4.2 Kernel Trick
		2.5 Classical Approaches for Image Classification
		2.6 Exercises
	3 Linear, Logistic, and Kernel Regression
		3.1 Introduction
		3.2 Linear Regression
			3.2.1 Ordinary Least Squares (OLS)
		3.3 Logistic Regression
			3.3.1 Logits and Linear Regression
			3.3.2 Multiclass Classification Using Logistic Regression
		3.4 Ridge Regression
		3.5 Kernel Regression
		3.6 Bias–Variance Trade-off in Regression
			3.6.1 Examples
		3.7 Exercises
	4 Reproducing Kernel Hilbert Space, Representer Theorem
		4.1 Introduction
		4.2 Reproducing Kernel Hilbert Space (RKHS)
			4.2.1 Feature Map and Kernels
			4.2.2 Definition of RKHS
		4.3 Representer Theorem
		4.4 Application of Representer Theorem
			4.4.1 Kernel Ridge Regression
			4.4.2 Kernel SVM
		4.5 Pros and Cons of Kernel Machines
		4.6 Exercises
Part II Building Blocks of Deep Learning
	5 Biological Neural Networks
		5.1 Introduction
		5.2 Neurons
			5.2.1 Anatomy of Neurons
			5.2.2 Signal Transmission Mechanism
			5.2.3 Synaptic Plasticity
		5.3 Biological Neural Network
			5.3.1 Visual System
			5.3.2 Hubel and Wiesel Model
			5.3.3 Jennifer Aniston Cell
		5.4 Exercises
	6 Artificial Neural Networks and Backpropagation
		6.1 Introduction
		6.2 Artificial Neural Networks
			6.2.1 Notation
			6.2.2 Modeling a Single Neuron
			6.2.3 Feedforward Multilayer ANN
		6.3 Artificial Neural Network Training
			6.3.1 Problem Formulation
			6.3.2 Optimizers
				6.3.2.1 Gradient Descent
				6.3.2.2 Stochastic Gradient Descent (SGD) Method
				6.3.2.3 Momentum Method
				6.3.2.4 Other Variations
		6.4 The Backpropagation Algorithm
			6.4.1 Derivation of the Backpropagation Algorithm
			6.4.2 Geometrical Interpretation of BP Algorithm
			6.4.3 Variational Interpretation of BP Algorithm
			6.4.4 Local Variational Formulation
		6.5 Exercises
	7 Convolutional Neural Networks
		7.1 Introduction
		7.2 History of Modern CNNs
			7.2.1 AlexNet
			7.2.2 GoogLeNet
			7.2.3 VGGNet
			7.2.4 ResNet
			7.2.5 DenseNet
			7.2.6 U-Net
		7.3 Basic Building Blocks of CNNs
			7.3.1 Convolution
			7.3.2 Pooling and Unpooling
			7.3.3 Skip Connection
		7.4 Training CNNs
			7.4.1 Loss Functions
			7.4.2 Data Split
			7.4.3 Regularization
				7.4.3.1 Data Augmentation
				7.4.3.2 Parameter Regularization
				7.4.3.3 Dropout
		7.5 Visualizing CNNs
		7.6 Applications of CNNs
		7.7 Exercises
	8 Graph Neural Networks
		8.1 Introduction
		8.2 Mathematical Preliminaries
			8.2.1 Definition
			8.2.2 Graph Isomorphism
			8.2.3 Graph Coloring
		8.3 Related Works
			8.3.1 Word Embedding
				8.3.1.1 CBOW
				8.3.1.2 Skip-Gram
			8.3.2 Loss Function
		8.4 Graph Embedding
			8.4.1 Matrix Factorization Approaches
			8.4.2 Random Walks Approaches
				8.4.2.1 DeepWalks
				8.4.2.2 Node2vec
			8.4.3 Neural Network Approaches
		8.5 WL Test, Graph Neural Networks
			8.5.1 Weisfeiler–Lehman Isomorphism Test
			8.5.2 Graph Neural Network as WL Test
		8.6 Summary and Outlook
		8.7 Exercises
	9 Normalization and Attention
		9.1 Introduction
			9.1.1 Notation
		9.2 Normalization
			9.2.1 Batch Normalization
			9.2.2 Layer and Instance Normalization
			9.2.3 Adaptive Instance Normalization (AdaIN)
			9.2.4 Whitening and Coloring Transform (WCT)
		9.3 Attention
			9.3.1 Metabotropic Receptors: Biological Analogy
			9.3.2 Mathematical Modeling of Spatial Attention
			9.3.3 Channel Attention
		9.4 Applications
			9.4.1 StyleGAN
			9.4.2 Self-Attention GAN
			9.4.3 Attentional GAN: Text to Image Generation
			9.4.4 Graph Attention Network
			9.4.5 Transformer
			9.4.6 BERT
			9.4.7 Generative Pre-trained Transformer (GPT)
			9.4.8 Vision Transformer
		9.5 Mathematical Analysis of Normalization and Attention
		9.6 Exercises
Part III Advanced Topics in Deep Learning
	10 Geometry of Deep Neural Networks
		10.1 Introduction
			10.1.1 Desiderata of Machine Learning
		10.2 Case Studies
			10.2.1 Single–Layer Perceptron
			10.2.2 Frame Representation
		10.3 Convolution Framelets
			10.3.1 Convolution and Hankel Matrix
			10.3.2 Convolution Framelet Expansion
			10.3.3 Link to CNN
			10.3.4 Deep Convolutional Framelets
		10.4 Geometry of CNN
			10.4.1 Role of Nonlinearity
			10.4.2 Nonlinearity Is the Key for Inductive Learning
			10.4.3 Expressivity
			10.4.4 Geometric Meaning of Features
			10.4.5 Geometric Understanding of Autoencoder
			10.4.6 Geometric Understanding of Classifier
		10.5 Open Problems
		10.6 Exercises
	11 Deep Learning Optimization
		11.1 Introduction
		11.2 Problem Formulation
		11.3 Polyak–Łojasiewicz-Type Convergence Analysis
			11.3.1 Loss Landscape and Over-Parameterization
		11.4 Lyapunov-Type Convergence Analysis
			11.4.1 The Neural Tangent Kernel (NTK)
			11.4.2 NTK at Infinite Width Limit
			11.4.3 NTK for General Loss Function
		11.5 Exercises
	12 Generalization Capability of Deep Learning
		12.1 Introduction
		12.2 Mathematical Preliminaries
			12.2.1 Vapnik–Chervonenkis (VC) Bounds
			12.2.2 Rademacher Complexity Bounds
			12.2.3 PAC–Bayes Bounds
		12.3 Reconciling the Generalization Gap via Double Descent Model
		12.4 Inductive Bias of Optimization
		12.5 Generalization Bounds via Algorithm Robustness
		12.6 Exercises
	13 Generative Models and Unsupervised Learning
		13.1 Introduction
		13.2 Mathematical Preliminaries
		13.3 Statistical Distances
			13.3.1 f-Divergence
				13.3.1.1 Kullback–Leibler (KL) Divergence
				13.3.1.2 Jensen–Shannon (JS) Divergence
			13.3.2 Wasserstein Metric
		13.4 Optimal Transport
			13.4.1 Monge\'s Original Formulation
			13.4.2 Kantorovich Formulation
			13.4.3 Entropy Regularization
		13.5 Generative Adversarial Networks
			13.5.1 Earliest Form of GAN
			13.5.2 f-GAN
			13.5.3 Wasserstein GAN (W-GAN)
			13.5.4 StyleGAN
		13.6 Autoencoder-Type Generative Models
			13.6.1 ELBO
			13.6.2 Variational Autoencoder (VAE)
			13.6.3 β-VAE
			13.6.4 Normalizing Flow, Invertible Flow
		13.7 Unsupervised Learning via Image Translation
			13.7.1 Pix2pix
			13.7.2 CycleGAN
			13.7.3 StarGAN
			13.7.4 Collaborative GAN
		13.8 Summary and Outlook
		13.9 Exercises
	14 Summary and Outlook
	15 Bibliography
Index