An Intuitive Exploration of Artificial Intelligence: Theory and Applications of Deep Learning

Preface
Contents
Acronyms
Author's Note
Part I Foundations
	1 AI Sculpture
		1.1 Manifolds in High Dimensions
		1.2 Sculpting Process
		1.3 Notational Convention
		1.4 Regression and Classification
			1.4.1 Linear Regression and Logistic Regression
			1.4.2 Regression Loss and Cross-Entropy Loss
			1.4.3 Sculpting with Shades
		1.5 Discriminative and Generative AI
		1.6 Success of Discriminative Methods
		1.7 Feature Engineering in Classical ML
		1.8 Supervised and Unsupervised AI
		1.9 Beyond Manifolds
		1.10 Chapter Summary
	2 Make Me Learn
		2.1 Learnable Parameters
			2.1.1 The Power of a Single Neuron
			2.1.2 Neurons Working Together
		2.2 Backpropagation of Gradients
			2.2.1 Partial Derivatives
			2.2.2 Forward and Backward Passes
		2.3 Stochastic Gradient Descent
			2.3.1 Handling Difficult Landscapes
			2.3.2 Stabilization of Training
		2.4 Chapter Summary
	3 Images and Sequences
		3.1 Convolutional Neural Networks
			3.1.1 The Biology of the Visual Cortex
			3.1.2 Pattern Matching
			3.1.3 3-D Convolution
		3.2 Recurrent Neural Networks
			3.2.1 Neurons with States
			3.2.2 The Power of Recurrence
			3.2.3 Going Both Ways
			3.2.4 Attention
		3.3 Self-Attention
		3.4 LSTM
		3.5 Beyond Images and Sequences
		3.6 Chapter Summary
	4 Why AI Works
		4.1 Convex Polytopes
		4.2 Piecewise Linear Function
			4.2.1 Subdivision of the Input Space
			4.2.2 Piecewise Non-linear Function
			4.2.3 Carving Out the Feature Spaces
		4.3 Expressive Power of AI
		4.4 Convolutional Neural Network
		4.5 Recurrent Neural Network
		4.6 Architectural Variations
		4.7 Attention and Carving
		4.8 Optimization Landscape
			4.8.1 Graph-Induced Polynomial
			4.8.2 Gradient of the Loss Function
			4.8.3 Visualization
			4.8.4 Critical Points
		4.9 The Mathematics of Loss Landscapes
			4.9.1 Random Polynomial Perspective
			4.9.2 Random Matrix Perspective
			4.9.3 Spin Glass Perspective
			4.9.4 Computational Complexity Perspective
			4.9.5 SGD and Critical Points
			4.9.6 Confluence of Perspectives
		4.10 Distributed Representation and Intrinsic Dimension
		4.11 Chapter Summary
	5 Novice to Maestro
		5.1 How AI Learns to Sculpt
			5.1.1 Training Data
			5.1.2 Evaluation Metrics
			5.1.3 Hyperparameter Search
			5.1.4 Regularization
			5.1.5 Bias and Variance
			5.1.6 A Fairy Tale in the Land of ML
		5.2 Learning Curves
		5.3 From the Lab to the Dirty Field
		5.4 System Design
		5.5 Flavors of Learning
		5.6 Ingenuity and Big Data in the Success of AI
		5.7 Chapter Summary
	6 Unleashing the Power of Generation
		6.1 Creating Universes
		6.2 To Recognize It, Learn to Draw It
		6.3 General Definition
		6.4 Generative Parameters
		6.5 Generative AI Models
			6.5.1 Restricted Boltzmann Machines
			6.5.2 Autoencoders
			6.5.3 Variational Autoencoder
			6.5.4 Pixel Recursive Models
			6.5.5 Generative Adversarial Networks
			6.5.6 Wasserstein Generative Adversarial Networks
		6.6 The Carving Process in Generative AI
		6.7 Representation of Individual Signals
		6.8 Chapter Summary
	7 The Road Most Rewarded
		7.1 Reinforcement Learning
		7.2 Learning an Optimal Policy
		7.3 Deep Q-Learning
		7.4 Policy Gradient
			7.4.1 Intuition
			7.4.2 Mathematical Analysis
		7.5 Let's Play and Explore
		7.6 Chapter Summary
	8 The Classical World
		8.1 Maximum Likelihood Estimation
		8.2 Uncertainty in Estimation
		8.3 Linear Models
			8.3.1 Linear Regression
			8.3.2 The Geometry of Linear Regression
			8.3.3 Regularization
			8.3.4 Logistic Regression
		8.4 Classical ML
			8.4.1 k-Nearest Neighbors
			8.4.2 Naive Bayes Classifier
			8.4.3 FDA, LDA, and QDA
			8.4.4 Support Vector Machines
			8.4.5 Neural Networks
			8.4.6 Decision Trees
			8.4.7 Gaussian Process Regression
			8.4.8 Unsupervised Methods
		8.5 XGBoost
			8.5.1 Relevance Ranking
		8.6 Chapter Summary
Part II Applications
	9 To See Is to Believe
		9.1 Image Classification
			9.1.1 Motivating Examples
			9.1.2 LeNet
			9.1.3 Stacked Autoencoders
			9.1.4 AlexNet
			9.1.5 VGG
			9.1.6 ResNet
			9.1.7 Inception V3
			9.1.8 Showroom of Models
			9.1.9 Your Own Network
		9.2 Object Detection as Classification
			9.2.1 Sliding Window Method
			9.2.2 Region Proposal Method
		9.3 Regression on Images
			9.3.1 Motivating Examples
			9.3.2 Object Detection as Regression
				9.3.2.1 Regression Output
				9.3.2.2 Grid-Based Approach
		9.4 Attention in Computer Vision
		9.5 Semantic Segmentation
		9.6 Image Similarity
		9.7 Video Analysis
		9.8 3-D Data
		9.9 Self-Driving Vehicles
		9.10 Present and Future
		9.11 Protein Folding
		9.12 Chapter Summary
	10 Read, Read, Read
		10.1 Natural Language Understanding
		10.2 Embedding Words in a Semantic Space
		10.3 Sequence to Sequence
			10.3.1 Encoder-Decoder Architecture
			10.3.2 Neural Machine Translation
		10.4 Attention Mechanism
		10.5 Self-Attention
		10.6 Creativity in NLU Solutions
		10.7 AI and Human Culture
		10.8 Recommender Systems
		10.9 Reward-Based Formulations
		10.10 Chapter Summary
	11 Lend Me Your Ear
		11.1 Classical Speech Recognition
		11.2 Spectrogram to Transcription
			11.2.1 Alignment-Free Temporal Connections
			11.2.2 End-to-End Solution
			11.2.3 Don't Listen to Others
		11.3 Speech Synthesis
		11.4 Handwriting Recognition
		11.5 Chapter Summary
	12 Create Your Shire and Rivendell
		12.1 From Neurons to Art
			12.1.1 DeepDream
			12.1.2 Style Transfer
		12.2 Image Translation
		12.3 DeepFake
		12.4 Creative Applications
		12.5 Chapter Summary
	13 Math to Code to Petaflops
		13.1 Software Frameworks
			13.1.1 The Twentieth Century
			13.1.2 The Twenty-First Century
			13.1.3 AI Frameworks
		13.2 Let's Crunch Numbers
			13.2.1 Computing Hardware
			13.2.2 GPU Machines
			13.2.3 Cloud GPU Instances
			13.2.4 Training Script
			13.2.5 Deployment
		13.3 Speeding Up Training
			13.3.1 Data Parallelism
			13.3.2 Delayed and Compressed SGD
		13.4 Open Ecosystem and Efficient Hardware
		13.5 Chapter Summary
	14 AI and Business
		14.1 Strategy
		14.2 Organization
		14.3 Execution
		14.4 Evaluation
		14.5 Startups
		14.6 Chapter Summary
Part III The Road Ahead
	15 Keep Marching On
		15.1 Robust AI
			15.1.1 Adversarial Examples
			15.1.2 Learning from Human Vision
			15.1.3 Fusion of Evidence
			15.1.4 Interpretable AI
		15.2 AI Extraordinaire
		15.3 Chapter Summary
	16 Benevolent AI for All
		16.1 Benefits of AI
		16.2 AI in Medicine
		16.3 Dangers of AI
		16.4 AI-Human Conflict
		16.5 Choices Ahead
		16.6 Chapter Summary
	17 Am I Looking at Myself?
		17.1 Is It Computable or Non-computable?
		17.2 Is Consciousness Everywhere?
		17.3 Who Is the Storyteller?
		17.4 Chapter Summary
A Solutions
	Answer of Exercise 1
	Answer of Exercise 2
	Answer of Exercise 3
	Answer of Exercise 4
	Answer of Exercise 5
	Answer of Exercise 6
	Answer of Exercise 7
	Answer of Exercise 9
		Answer of Exercise 8
	Answer of Exercise 10
	Answer of Exercise 11
	Answer of Exercise 12
	Answer of Exercise 13
	Answer of Exercise 14
	Answer of Exercise 15
	Answer of Exercise 16
	Answer of Exercise 17
	Answer of Exercise 18
	Answer of Exercise 19
	Answer of Exercise 20
	Answer of Exercise 21
	Answer of Exercise 22
	Answer of Exercise 23
	Answer of Exercise 24
	Answer of Exercise 25
	Answer of Exercise 26
	Answer of Exercise 27
	Answer of Exercise 28
	Answer of Exercise 29
	Answer of Exercise 30
	Answer of Exercise 31
	Answer of Exercise 32
	Answer of Exercise 33
	Answer of Exercise 34
	Answer of Exercise 35
B Lab Exercises and Projects
	B.1 Exercises
	B.2 Exploration
	B.3 Debate and Discussion
Further Reading
Glossary
References
Index