Deep Learning with TensorFlow and Keras: Build and deploy supervised, unsupervised, deep, and reinforcement learning models, 3rd Edition

Cover
Copyright
Contributors
Table of Contents
Preface
Chapter 1: Neural Network Foundations with TF
	What is TensorFlow (TF)?
	What is Keras?
	Introduction to neural networks
	Perceptron
		Our first example of TensorFlow code
	Multi-layer perceptron: our first example of a network
		Problems in training the perceptron and solution
		Activation function: sigmoid
		Activation function: tanh
		Activation function: ReLU
		Two additional activation functions: ELU and Leaky ReLU
		Activation functions
		In short: what are neural networks after all?
	A real example: recognizing handwritten digits
		One hot-encoding (OHE)
		Defining a simple neural net in TensorFlow
		Running a simple TensorFlow net and establishing a baseline
		Improving the simple net in TensorFlow with hidden layers
		Further improving the simple net in TensorFlow with dropout
		Testing different optimizers in TensorFlow
		Increasing the number of epochs
		Controlling the optimizer learning rate
		Increasing the number of internal hidden neurons
		Increasing the size of batch computation
		Summarizing experiments run to recognizing handwritten digits
	Regularization
		Adopting regularization to avoid overfitting
		Understanding batch normalization
	Playing with Google Colab: CPUs, GPUs, and TPUs
	Sentiment analysis
		Hyperparameter tuning and AutoML
	Predicting output
	A practical overview of backpropagation
	What have we learned so far?
	Toward a deep learning approach
	Summary
	References
Chapter 2: Regression and Classification
	What is regression?
	Prediction using linear regression
		Simple linear regression
		Multiple linear regression
		Multivariate linear regression
	Neural networks for linear regression
		Simple linear regression using TensorFlow Keras
		Multiple and multivariate linear regression using the TensorFlow Keras API
	Classification tasks and decision boundaries
		Logistic regression
		Logistic regression on the MNIST dataset
	Summary
	References
Chapter 3: Convolutional Neural Networks
	Deep convolutional neural networks
		Local receptive fields
		Shared weights and bias
		A mathematical example
		ConvNets in TensorFlow
		Pooling layers
			Max pooling
			Average pooling
		ConvNets summary
	An example of DCNN: LeNet
		LeNet code in TF
		Understanding the power of deep learning
	Recognizing CIFAR-10 images with deep learning
		Improving the CIFAR-10 performance with a deeper network
		Improving the CIFAR-10 performance with data augmentation
		Predicting with CIFAR-10
	Very deep convolutional networks for large-scale image recognition
		Recognizing cats with a VGG16 net work
		Utilizing the tf.Keras built-in VGG16 net module
		Recycling pre-built deep learning models for extracting features
	Deep Inception V3 for transfer learning
	Other CNN architectures
		AlexNet
		Residual networks
		HighwayNets and DenseNets
		Xception
	Style transfer
		Content distance
		Style distance
	Summary
	References
Chapter 4: Word Embeddings
	Word embedding ‒ origins and fundamentals
	Distributed representations
	Static embeddings
		Word2Vec
		GloVe
	Creating your own embeddings using Gensim
	Exploring the embedding space with Gensim
	Using word embeddings for spam detection
		Getting the data
		Making the data ready for use
		Building the embedding matrix
		Defining the spam classifier
		Training and evaluating the model
		Running the spam detector
	Neural embeddings – not just for words
		Item2Vec
		node2vec
	Character and subword embeddings
	Dynamic embeddings
	Sentence and paragraph embeddings
	Language model-based embeddings
		Using BERT as a feature extractor
	Summary
	References
Chapter 5: Recurrent Neural Networks
	The basic RNN cell
		Backpropagation through time (BPTT)
		Vanishing and exploding gradients
	RNN cell variants
		Long short-term memory (LSTM)
		Gated recurrent unit (GRU)
		Peephole LSTM
	RNN variants
		Bidirectional RNNs
		Stateful RNNs
	RNN topologies
		Example ‒ One-to-many – Learning to generate text
		Example ‒ Many-to-one – Sentiment analysis
		Example ‒ Many-to-many – POS tagging
	Encoder-decoder architecture – seq2seq
		Example ‒ seq2seq without attention for machine translation
	Attention mechanism
		Example ‒ seq2seq with attention for machine translation
	Summary
	References
Chapter 6: Transformers
	Architecture
		Key intuitions
			Positional encoding
			Attention
			Self-attention
			Multi-head (self-)attention
		How to compute attention
		Encoder-decoder architecture
		Residual and normalization layers
		An overview of the transformer architecture
		Training
	Transformers’ architectures
		Categories of transformers
			Decoder or autoregressive
			Encoder or autoencoding
			Seq2seq
			Multimodal
			Retrieval
		Attention
			Full versus sparse
			LSH attention
			Local attention
	Pretraining
		Encoder pretraining
		Decoder pretraining
		Encoder-decoder pretraining
		A taxonomy for pretraining tasks
	An overview of popular and well-known models
		BERT
		GPT-2
		GPT-3
		Reformer
		BigBird
		Transformer-XL
		XLNet
		RoBERTa
		ALBERT
		StructBERT
		T5 and MUM
		ELECTRA
		DeBERTa
		The Evolved Transformer and MEENA
		LaMDA
		Switch Transformer
		RETRO
		Pathways and PaLM
	Implementation
		Transformer reference implementation: An example of translation
		Hugging Face
			Generating text
			Autoselecting a model and autotokenization
			Named entity recognition
			Summarization
			Fine-tuning
		TFHub
	Evaluation
		Quality
			GLUE
			SuperGLUE
			SQuAD
			RACE
			NLP-progress
		Size
			Larger doesn’t always mean better
		Cost of serving
	Optimization
		Quantization
		Weight pruning
		Distillation
	Common pitfalls: dos and don’ts
		Dos
		Don’ts
	The future of transformers
	Summary
Chapter 7: Unsupervised Learning
	Principal component analysis
		PCA on the MNIST dataset
		TensorFlow Embedding API
	K-means clustering
		K-means in TensorFlow
		Variations in k-means
	Self-organizing maps
		Colour mapping using a SOM
	Restricted Boltzmann machines
		Reconstructing images using an RBM
		Deep belief networks
	Summary
	References
Chapter 8: Autoencoders
	Introduction to autoencoders
	Vanilla autoencoders
		TensorFlow Keras layers ‒ defining custom layers
		Reconstructing handwritten digits using an autoencoder
	Sparse autoencoder
	Denoising autoencoders
		Clearing images using a denoising autoencoder
	Stacked autoencoder
		Convolutional autoencoder for removing noise from images
		A TensorFlow Keras autoencoder example ‒ sentence vectors
	Variational autoencoders
	Summary
	References
Chapter 9: Generative Models
	What is a GAN?
		MNIST using GAN in TensorFlow
	Deep convolutional GAN (DCGAN)
		DCGAN for MNIST digits
	Some interesting GAN architectures
		SRGAN
		CycleGAN
		InfoGAN
	Cool applications of GANs
	CycleGAN in TensorFlow
	Flow-based models for data generation
	Diffusion models for data generation
	Summary
	References
Chapter 10: Self-Supervised Learning
	Previous work
	Self-supervised learning
	Self-prediction
		Autoregressive generation
			PixelRNN
			Image GPT (IPT)
			GPT-3
			XLNet
			WaveNet
			WaveRNN
		Masked generation
			BERT
			Stacked denoising autoencoder
			Context autoencoder
			Colorization
		Innate relationship prediction
			Relative position
			Solving jigsaw puzzles
			Rotation
		Hybrid self-prediction
			VQ-VAE
			Jukebox
			DALL-E
			VQ-GAN
	Contrastive learning
		Training objectives
			Contrastive loss
			Triplet loss
			N-pair loss
			Lifted structural loss
			NCE loss
			InfoNCE loss
			Soft nearest neighbors loss
		Instance transformation
			SimCLR
			Barlow Twins
			BYOL
			Feature clustering
			DeepCluster
			SwAV
			InterCLR
		Multiview coding
			AMDIM
			CMC
		Multimodal models
			CLIP
			CodeSearchNet
			Data2Vec
	Pretext tasks
	Summary
	References
Chapter 11: Reinforcement Learning
	An introduction to RL
		RL lingo
		Deep reinforcement learning algorithms
			How does the agent choose its actions, especially when untrained?
			How does the agent maintain a balance between exploration and exploitation?
			How to deal with the highly correlated input state space
			How to deal with the problem of moving targets
		Reinforcement success in recent years
	Simulation environments for RL
	An introduction to OpenAI Gym
		Random agent playing Breakout
		Wrappers in Gym
	Deep Q-networks
		DQN for CartPole
		DQN to play a game of Atari
		DQN variants
			Double DQN
			Dueling DQN
			Rainbow
	Deep deterministic policy gradient
	Summary
	References
Chapter 12: Probabilistic TensorFlow
	TensorFlow Probability
	TensorFlow Probability distributions
		Using TFP distributions
			Coin Flip Example
			Normal distribution
		Bayesian networks
		Handling uncertainty in predictions using TensorFlow Probability
			Aleatory uncertainty
			Epistemic uncertainty
			Creating a synthetic dataset
			Building a regression model using TensorFlow
			Probabilistic neural networks for aleatory uncertainty
			Accounting for the epistemic uncertainty
	Summary
	References
Chapter 13: An Introduction to AutoML
	What is AutoML?
	Achieving AutoML
	Automatic data preparation
	Automatic feature engineering
	Automatic model generation
	AutoKeras
	Google Cloud AutoML and Vertex AI
		Using the Google Cloud AutoML Tables solution
		Using the Google Cloud AutoML Text solution
		Using the Google Cloud AutoML Video solution
		Cost
	Summary
	References
Chapter 14: The Math Behind Deep Learning
	History
	Some mathematical tools
		Vectors
		Derivatives and gradients everywhere
		Gradient descent
		Chain rule
		A few differentiation rules
		Matrix operations
	Activation functions
		Derivative of the sigmoid
		Derivative of tanh
		Derivative of ReLU
	Backpropagation
		Forward step
		Backstep
			Case 1: From hidden layer to output layer
			Case 2: From hidden layer to hidden layer
		Cross entropy and its derivative
		Batch gradient descent, stochastic gradient descent, and mini-batch
			Batch gradient descent
			Stochastic gradient descent
			Mini-batch gradient descent
		Thinking about backpropagation and ConvNets
		Thinking about backpropagation and RNNs
	A note on TensorFlow and automatic differentiation
	Summary
	References
Chapter 15: Tensor Processing Unit
	C/G/T processing units
		CPUs and GPUs
		TPUs
	Four generations of TPUs, plus Edge TPU
		First generation TPU
		Second generation TPU
		Third generation TPU
		Fourth generation TPUs
		Edge TPU
	TPU performance
	How to use TPUs with Colab
		Checking whether TPUs are available
		Keras MNIST TPU end-to-end training
	Using pretrained TPU models
	Summary
	References
Chapter 16: Other Useful Deep Learning Libraries
	Hugging Face
	OpenAI
		OpenAI GPT-3 API
		OpenAI DALL-E 2
		OpenAI Codex
	PyTorch
	ONNX
	H2O.ai
		H2O AutoML
		AutoML using H2O
		H2O model explainability
			Partial dependence plots
			Variable importance heatmap
			Model correlation
	Summary
Chapter 17: Graph Neural Networks
	Graph basics
	Graph machine learning
	Graph convolutions – the intuition behind GNNs
	Common graph layers
		Graph convolution network
		Graph attention network
		GraphSAGE (sample and aggregate)
		Graph isomorphism network
	Common graph applications
		Node classification
		Graph classification
		Link prediction
	Graph customizations
		Custom layers and message passing
		Custom graph dataset
			Single graphs in datasets
			Set of multiple graphs in datasets
	Future directions
		Heterogeneous graphs
		Temporal Graphs
	Summary
	References
Chapter 18: Machine Learning Best Practices
	The need for best practices
	Data best practices
		Feature selection
		Features and data
			Augmenting textual data
	Model best practices
		Baseline models
		Pretrained models, model APIs, and AutoML
		Model evaluation and validation
		Model improvements
	Summary
	References
Chapter 19: TensorFlow 2 Ecosystem
	TensorFlow Hub
		Using pretrained models for inference
	TensorFlow Datasets
		Load a TFDS dataset
		Building data pipelines using TFDS
	TensorFlow Lite
		Quantization
		FlatBuffers
		Mobile converter
		Mobile optimized interpreter
		Supported platforms
		Architecture
		Using TensorFlow Lite
		A generic example of an application
		Using GPUs and accelerators
		An example of an application
	Pretrained models in TensorFlow Lite
		Image classification
		Object detection
		Pose estimation
		Smart reply
		Segmentation
		Style transfer
		Text classification
		Large language models
		A note about using mobile GPUs
	An overview of federated learning at the edge
		TensorFlow FL APIs
	TensorFlow.js
		Vanilla TensorFlow.js
		Converting models
		Pretrained models
		Node.js
	Summary
	References
Chapter 20: Advanced Convolutional Neural Networks
	Composing CNNs for complex tasks
		Classification and localization
		Semantic segmentation
		Object detection
		Instance segmentation
		Keras Applications
		TensorFlow Hub
	Answering questions about images (visual Q&A)
	Creating a DeepDream network
	Inspecting what a network has learned
	Video
		Classifying videos with pretrained nets in six different ways
	Text documents
		Using a CNN for sentiment analysis
	Audio and music
		Dilated ConvNets, WaveNet, and NSynth
	A summary of convolution operations
		Basic CNNs
		Dilated convolution
		Transposed convolution
		Separable convolution
		Depthwise convolution
		Depthwise separable convolution
	Capsule networks
		What is the problem with CNNs?
		What is new with capsule networks?
	Summary
	References
Other Books You May Enjoy
Index