Deep Learning Patterns and Practices

Deep Learning Patterns and Practices
brief contents
contents
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized: A roadmap
	About the code
	liveBook discussion forum
	Other online resources
about the author
about the cover illustration
Part 1: Deep learning fundamentals
	Chapter 1: Designing modern machine learning
		1.1 A focus on adaptability
			1.1.1 Computer vision leading the way
			1.1.2 Beyond computer vision: NLP, NLU, structured data
		1.2 The evolution in machine learning approaches
			1.2.1 Classical AI vs. narrow AI
			1.2.2 Next steps in computer learning
		1.3 The benefits of design patterns
	Chapter 2: Deep neural networks
		2.1 Neural network basics
			2.1.1 Input layer
			2.1.2 Deep neural networks
			2.1.3 Feed-forward networks
			2.1.4 Sequential API method
			2.1.5 Functional API method
			2.1.6 Input shape vs. input layer
			2.1.7 Dense layer
			2.1.8 Activation functions
			2.1.9 Shorthand syntax
			2.1.10 Improving accuracy with an optimizer
		2.2 DNN binary classifier
		2.3 DNN multiclass classifier
		2.4 DNN multilabel multiclass classifier
		2.5 Simple image classifier
			2.5.1 Flattening
			2.5.2 Overfitting and dropout
	Chapter 3: Convolutional and residual neural networks
		3.1 Convolutional neural networks
			3.1.1 Why we use a CNN over a DNN for image models
			3.1.2 Downsampling (resizing)
			3.1.3 Feature detection
			3.1.4 Pooling
			3.1.5 Flattening
		3.2 The ConvNet design for a CNN
		3.3 VGG networks
		3.4 ResNet networks
			3.4.1 Architecture
			3.4.2 Batch normalization
			3.4.3 ResNet50
	Chapter 4: Training fundamentals
		4.1 Forward feeding and backward propagation
			4.1.1 Feeding
			4.1.2 Backward propagation
		4.2 Dataset splitting
			4.2.1 Training and test sets
			4.2.2 One-hot encoding
		4.3 Data normalization
			4.3.1 Normalization
			4.3.2 Standardization
		4.4 Validation and overfitting
			4.4.1 Validation
			4.4.2 Loss monitoring
			4.4.3 Going deeper with layers
		4.5 Convergence
		4.6 Checkpointing and early stopping
			4.6.1 Checkpointing
			4.6.2 Early stopping
		4.7 Hyperparameters
			4.7.1 Epochs
			4.7.2 Steps
			4.7.3 Batch size
			4.7.4 Learning rate
		4.8 Invariance
			4.8.1 Translational invariance
			4.8.2 Scale invariance
			4.8.3 TF.Keras ImageDataGenerator
		4.9 Raw (disk) datasets
			4.9.1 Directory structure
			4.9.2 CSV file
			4.9.3 JSON file
			4.9.4 Reading images
			4.9.5 Resizing
		4.10 Model save/restore
			4.10.1 Save
			4.10.2 Restore
Part 2: Basic design pattern
	Chapter 5: Procedural design pattern
		5.1 Basic neural network architecture
		5.2 Stem component
			5.2.1 VGG
			5.2.2 ResNet
			5.2.3 ResNeXt
			5.2.4 Xception
		5.3 Pre-stem
		5.4 Learner component
			5.4.1 ResNet
			5.4.2 DenseNet
		5.5 Task component
			5.5.1 ResNet
			5.5.2 Multilayer output
			5.5.3 SqueezeNet
		5.6 Beyond computer vision: NLP
			5.6.1 Natural-language understanding
			5.6.2 Transformer architecture
	Chapter 6: Wide convolutional neural networks
		Inception V1
		6.1 Inception v1
			6.1.1 Naive inception module
			6.1.2 Inception v1 module
			6.1.3 Stem
			6.1.4 Learner
			6.1.5 Auxiliary classifiers
			6.1.6 Classifier
		6.2 Inception v2: Factoring convolutions
		6.3 Inception v3: Architecture redesign
			6.3.1 Inception groups and blocks
			6.3.2 Normal convolution
			6.3.3 Spatial separable convolution
			6.3.4 Stem redesign and implementation
			6.3.5 Auxiliary classifier
		6.4 ResNeXt: Wide residual neural networks
			6.4.1 ResNeXt block
			6.4.2 ResNeXt architecture
		6.5 Wide residual network
			6.5.1 WRN-50-2 architecture
			6.5.2 Wide residual block
		6.6 Beyond computer vision: Structured data
	Chapter 7: Alternative connectivity patterns
		7.1 DenseNet: Densely connected convolutional neural network
			7.1.1 Dense group
			7.1.2 Dense block
			7.1.3 DenseNet macro-architecture
			7.1.4 Dense transition block
		7.2 Xception: Extreme Inception
			7.2.1 Xception architecture
			7.2.2 Entry flow of Xception
			7.2.3 Middle flow of Xception
			7.2.4 Exit flow of Xception
			7.2.5 Depthwise separable convolution
			7.2.6 Depthwise convolution
			7.2.7 Pointwise convolution
		7.3 SE-Net: Squeeze and excitation
			7.3.1 Architecture of SE-Net
			7.3.2 Group and block of SE-Net
			7.3.3 SE link
	Chapter 8: Mobile convolutional neural networks
		8.1 MobileNet v1
			8.1.1 Architecture
			8.1.2 Width multiplier
			8.1.3 Resolution multiplier
			8.1.4 Stem
			8.1.5 Learner
			8.1.6 Classifier
		8.2 MobileNet v2
			8.2.1 Architecture
			8.2.2 Stem
			8.2.3 Learner
			8.2.4 Classifier
		8.3 SqueezeNet
			8.3.1 Architecture
			8.3.2 Stem
			8.3.3 Learner
			8.3.4 Classifier
			8.3.5 Bypass connections
		8.4 ShuffleNet v1
			8.4.1 Architecture
			8.4.2 Stem
			8.4.3 Learner
		8.5 Deployment
			8.5.1 Quantization
			8.5.2 TF Lite conversion and prediction
	Chapter 9: Autoencoders
		9.1 Deep neural network autoencoders
			9.1.1 Autoencoder architecture
			9.1.2 Encoder
			9.1.3 Decoder
			9.1.4 Training
		9.2 Convolutional autoencoders
			9.2.1 Architecture
			9.2.2 Encoder
			9.2.3 Decoder
		9.3 Sparse autoencoders
		9.4 Denoising autoencoders
		9.5 Super-resolution
			9.5.1 Pre-upsampling SR
			9.5.2 Post-upsampling SR
		9.6 Pretext tasks
		9.7 Beyond computer vision: sequence to sequence
Part 3: Working with pipelines
	Chapter 10: Hyperparameter tuning
		10.1 Weight initialization
			10.1.1 Weight distributions
			10.1.2 Lottery hypothesis
			10.1.3 Warm-up (numerical stability)
		10.2 Hyperparameter search fundamentals
			10.2.1 Manual method for hyperparameter search
			10.2.2 Grid search
			10.2.3 Random search
			10.2.4 KerasTuner
		10.3 Learning rate scheduler
			10.3.1 Keras decay parameter
			10.3.2 Keras learning rate scheduler
			10.3.3 Ramp
			10.3.4 Constant step
			10.3.5 Cosine annealing
		10.4 Regularization
			10.4.1 Weight regularization
			10.4.2 Label smoothing
		10.5 Beyond computer vision
	Chapter 11: Transfer learning
		11.1 TF.Keras prebuilt models
			11.1.1 Base model
			11.1.2 Pretrained ImageNet models for prediction
			11.1.3 New classifier
		11.2 TF Hub prebuilt models
			11.2.1 Using TF Hub pretrained models
			11.2.2 New classifier
		11.3 Transfer learning between domains
			11.3.1 Similar tasks
			11.3.2 Distinct tasks
			11.3.3 Domain-specific weights
			11.3.4 Domain transfer weight initialization
			11.3.5 Negative transfer
		11.4 Beyond computer vision
	Chapter 12: Data distributions
		12.1 Distribution types
			12.1.1 Population distribution
			12.1.2 Sampling distribution
			12.1.3 Subpopulation distribution
		12.2 Out of distribution
			12.2.1 The MNIST curated dataset
			12.2.2 Setting up the environment
			12.2.3 The challenge (“in the wild”)
			12.2.4 Training as a DNN
			12.2.5 Training as a CNN
			12.2.6 Image augmentation
			12.2.7 Final test
	Chapter 13: Data pipeline
		13.1 Data formats and storage
			13.1.1 Compressed and raw-image formats
			13.1.2 HDF5 format
			13.1.3 DICOM format
			13.1.4 TFRecord format
		13.2 Data feeding
			13.2.1 NumPy
			13.2.2 TFRecord
		13.3 Data preprocessing
			13.3.1 Preprocessing with a pre-stem
			13.3.2 Preprocessing with TF Extended
		13.4 Data augmentation
			13.4.1 Invariance
			13.4.2 Augmentation with tf.data
			13.4.3 Pre-stem
	Chapter 14: Training and deployment pipeline
		14.1 Model feeding
			14.1.1 Model feeding with tf.data.Dataset
			14.1.2 Distributed feeding with tf.Strategy
			14.1.3 Model feeding with TFX
		14.2 Training schedulers
			14.2.1 Pipeline versioning
			14.2.2 Metadata
			14.2.3 History
		14.3 Model evaluations
			14.3.1 Candidate vs. blessed model
			14.3.2 TFX evaluation
		14.4 Serving predictions
			14.4.1 On-demand (live) serving
			14.4.2 Batch prediction
			14.4.3 TFX pipeline components for deployment
			14.4.4 A/B testing
			14.4.5 Load balancing
			14.4.6 Continuous evaluation
		14.5 Evolution in production pipeline design
			14.5.1 Machine learning as a pipeline
			14.5.2 Machine learning as a CI/CD production process
			14.5.3 Model amalgamation in production
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X
	Z