Convolutional Neural Networks with Swift for Tensorflow

Table of Contents
About the Author
About the Technical Reviewer
Introduction
How this book is organized
Chapter 1: MNIST: 1D Neural Network
	Dataset overview
	Dataset handler
	Code: Multilayer perceptron + MNIST
	Results
	Demo breakdown (high level)
	Imports (1)
	Model breakdown (2)
	Global variables (3)
	Training loop: Updates (4)
	Training loop: Accuracy (5)
	Demo breakdown (low level)
	Fully connected neural network layers
	How the optimizer works
	Optimizers + neural networks
	Swift for Tensorflow
	Side quests
	Recap
Chapter 2: MNIST: 2D Neural Network
	Convolutions
		3x3 additive blur example
		3x3 Gaussian blur example
		Combined 3x3 convolutions – Sobel filter example
		3x3 striding
		Padding
		Maxpool
	2D MNIST model
		Code
		Side quest
	Recap
Chapter 3: CIFAR: 2D Neural Network with Blocks
	CIFAR dataset
	Color
	Breakdown
	Code
	Results
	Side quest
	Recap
Chapter 4: VGG Network
	Background: ImageNet
		Getting ImageNet
		Imagenette dataset
		Data augmentation
	VGG
	Code
		Results
		Memory usage
	Model refactoring
		VGG16 with subblocks
		Side quests
	Recap
Chapter 5: ResNet 34
	Skip connections
		Noise
		Batch normalization
	Code
		Results
		Side quest
	Recap
Chapter 6: ResNet 50
	Bottleneck blocks
	Code
	Results
	Side Quest: ImageNet
	Recap
Chapter 7: SqueezeNet
	SqueezeNet
		Fire modules
	Deep compression
		Model pruning
		Model quantization
		Size metric
		Difference between SqueezeNet 1.0 and 1.1
	Code
		Training loop
		Results
		Side quest
	Recap
Chapter 8: MobileNet v1
	MobileNet (v1)
		Spatial separable convolutions
		Depthwise convolutions
		Pointwise convolutions
	ReLU 6
		Example of the reduction in MACs with  this approach
	Code
		Results
	Recap
Chapter 9: MobileNet v2
	Inverted residual blocks
	Inverted skip connections
	Linear bottleneck layers
	Code
		Results
	Recap
Chapter 10: EfficientNet
	Swish
	SE (Squeeze + Excitation) block
	Code
	Results
		EfficientNet variants
		EfficientNet [B1-8]
		RandAugment
		Noisy Student
		EfficientDet
	Recap
Chapter 11: MobileNetV3
	Hard swish and hard sigmoid
	Remove the Squeeze and Excitation (SE) block logic for half the network
	Custom head
	Hyperparameters
	Performance
	Code
		Results
			EfficientNet-EdgeTPU
	Recap
Chapter 12: Bag of Tricks
	Bag of tricks
		What to learn from this
	Reading papers
		Stay behind the curve
		How I read papers
	Recap
Chapter 13: MNIST Revisited
	Next steps
	Pain points
	TPU case study
	Tensorflow 1 + Pytorch
	Enter functional programming
	Swift + TPU demo
	Results
	Recap
Chapter 14: You Are Here
	A (short and opinionated) history of computing
		History of GPUs
		Cloud computing
	Crossing the chasm
		Computer vision
		Direct applications
		Indirect applications
	Natural language processing
	Reinforcement learning and GANs
	Simulations in general
	To infinity and beyond
	Why Swift
	Why LLVM
	Why MLIR
	Why ML is the most important field
	Why now
	Why you
Appendix A: Cloud Setup
	Outline
	Google Cloud with CPU instances
		How to sign up for Google Cloud
		Creating your first few instances
	Google Cloud with preconfigured GPU instance
	Google Cloud nits
		Cattle, not pets
		Basic Google Cloud nomenclature
			* Machine types
			* Buckets
			* Billing
		Cleaning up
	Recap
Appendix B: Hardware Prerequisites, Software Installation Guidelines, and Unix Quickstart
	Hardware
		Don’t go alone!
		GPU
			GPUs to buy
		Multiple GPUs
		CPU
			Motherboard
			PSU
			Cooling
		RAM
		SSD
		Recommendations
			Long term
			Some real-world usage examples
		Hardware recap
	Installing Ubuntu
		General prep
			OS install prep
			Download Ubuntu + flash to USB key
		OS install
			Extra screen
			Reboot
			Doing a sanity check of your new server
		Ubuntu recap
	Installing swift for tensorflow
		Installing graphics card drivers and swift for tensorflow
			CUDA 10.2 install process
			Installing cudnn
			Installing swift for tensorflow using prebuilt packages
				Download swift
			Python
			Verify you\'re using a GPU
			Autoencoder demo
			Reinforcement learning demo
		Swift for Tensorflow recap
	Installing s4tf from scratch
		There be dragons here
			How to build swift for tensorflow from scratch
				Prerequisites
				Installing cmake
				Packages we need
					Bazel
			Fetch swift for tensorflow sources
			What a checkout will look like (different hashes)
			Python 2 install + packages needed
			Build swift for tensorflow from source with GPU support
			Running our swift binary
			Reset your build artifacts
		Installing s4tf from scratch recap
	Client setup process + Unix quickstart
		Setting up your client computer/crash course in Unix
		General config
		Configuring your network for remote access
			Setting up a VPN (ideal but more complicated)
			Setting up port forwarding
		Crash course in tmux
Appendix C: Additional Resources
	Python --> swift transition guide
		Python 3
		REPL
		Python --> Swift bridge
		Python --> C bridge
		Python libraries
	Self-study guide
		Things to study
			Python
			Swift
			iOS/Android
			Tensorflow
			Pytorch
			fast.ai
			Cloud computing
			TPU
			Unix
			Git + Unix + etc
			Other machine learning compiler–related projects
		System monitoring/utilities
			Check standard system utilities
Index