Deep Learning with TensorFlow 2.0 and Keras: Regression, ConvNets, GANs, RNNs, NLP & more with TF 2.0 and the Keras API

Cover
Copyright
Packt Page
Contributors
Table of Contents
Preface
Chapter 1: Neural Network Foundations with TensorFlow 2.0
	What is TensorFlow (TF)?
	What is Keras?
	What are the most important changes in TensorFlow 2.0?
	Introduction to neural networks
	Perceptron
		A first example of TensorFlow 2.0 code
	Multi-layer perceptron – our first example of a network
		Problems in training the perceptron and their solutions
		Activation function – sigmoid
		Activation function – tanh
		Activation function – ReLU
		Two additional activation functions – ELU and LeakyReLU
		Activation functions
		In short – what are neural networks after all?
	A real example – recognizing handwritten digits
		One-hot encoding (OHE)
		Defining a simple neural network in TensorFlow 2.0
		Running a simple TensorFlow 2.0 net and establishing a baseline
		Improving the simple net in TensorFlow 2.0 with hidden layers
		Further improving the simple net in TensorFlow with Dropout
		Testing different optimizers in TensorFlow 2.0
		Increasing the number of epochs
		Controlling the optimizer learning rate
		Increasing the number of internal hidden neurons
		Increasing the size of batch computation
		Summarizing experiments run for recognizing handwritten charts
	Regularization
		Adopting regularization to avoid overfitting
		Understanding BatchNormalization
	Playing with Google Colab – CPUs, GPUs, and TPUs
	Sentiment analysis
	Hyperparameter tuning and AutoML
	Predicting output
	A practical overview of backpropagation
	What have we learned so far?
	Towards a deep learning approach
	References
Chapter 2: TensorFlow 1.x and 2.x
	Understanding TensorFlow 1.x
		TensorFlow 1.x computational graph program structure
			Computational graphs
		Working with constants, variables, and placeholders
		Examples of operations
			Constants
			Sequences
			Random tensors
			Variables
		An example of TensorFlow 1.x in TensorFlow 2.x
	Understanding TensorFlow 2.x
		Eager execution
		AutoGraph
		Keras APIs – three programming models
			Sequential API
			Functional API
			Model subclassing
		Callbacks
		Saving a model and weights
		Training from tf.data.datasets
		tf.keras or Estimators?
		Ragged tensors
		Custom training
		Distributed training in TensorFlow 2.x
			Multiple GPUs
			MultiWorkerMirroredStrategy
			TPUStrategy
			ParameterServerStrategy
		Changes in namespaces
		Converting from 1.x to 2.x
		Using TensorFlow 2.x effectively
	The TensorFlow 2.x ecosystem
		Language bindings
	Keras or tf.keras?
	Summary
Chapter 3: Regression
	What is regression?
	Prediction using linear regression
		Simple linear regression
		Multiple linear regression
		Multivariate linear regression
	TensorFlow Estimators
		Feature columns
		Input functions
		MNIST using TensorFlow Estimator API
	Predicting house price using linear regression
	Classification tasks and decision boundaries
		Logistic regression
		Logistic regression on the MNIST dataset
	Summary
	References
Chapter 4: Convolutional Neural Networks
	Deep Convolutional Neural Network (DCNN)
		Local receptive fields
		Shared weights and bias
		A mathematical example
		Convnets in TensorFlow 2.x
		Pooling layers
			Max pooling
			Average pooling
			ConvNets summary
	An example of DCNN ‒ LeNet
		LeNet code in TensorFlow 2.0
		Understanding the power of deep learning
	Recognizing CIFAR-10 images with deep learning
		Improving the CIFAR-10 performance with a deeper network
		Improving the CIFAR-10 performance with data augmentation
		Predicting with CIFAR-10
	Very deep convolutional networks for large-scale image recognition
		Recognizing cats with a VGG16 Net
		Utilizing tf.Keras built-in VGG16 Net module
		Recycling prebuilt deep learning models for extracting features
	Summary
	References
Chapter 5: Advanced Convolutional Neural Networks
	Computer vision
		Composing CNNs for complex tasks
			Classification and localization
			Semantic segmentation
			Object detection
			Instance segmentation
		Classifying Fashion-MNIST with a tf.keras - estimator model
		Run Fashion-MNIST the tf.keras - estimator model on GPUs
		Deep Inception-v3 Net used for transfer learning
		Transfer learning for classifying horses and humans
		Application Zoos with tf.keras and TensorFlow Hub
			Keras applications
			TensorFlow Hub
		Other CNN architectures
			AlexNet
			Residual networks
			HighwayNets and DenseNets
			Xception
		Answering questions about images (VQA)
		Style transfer
			Content distance
			Style distance
		Creating a DeepDream network
		Inspecting what a network has learned
	Video
		Classifying videos with pretrained nets in six different ways
	Textual documents
		Using a CNN for sentiment analysis
	Audio and music
		Dilated ConvNets, WaveNet, and NSynth
	A summary of convolution operations
		Basic convolutional neural networks (CNN or ConvNet)
		Dilated convolution
			Transposed convolution
		Separable convolution
		Depthwise convolution
		Depthwise separable convolution
	Capsule networks
		So what is the problem with CNNs?
		So what is new with Capsule networks?
	Summary
	References
Chapter 6: Generative Adversarial Networks
	What is a GAN?
		MNIST using GAN in TensorFlow
	Deep convolutional GAN (DCGAN)
		DCGAN for MNIST digits
	Some interesting GAN architectures
		SRGAN
		CycleGAN
		InfoGAN
	Cool applications of GANs
	CycleGAN in TensorFlow 2.0
	Summary
	References
Chapter 7: Word Embeddings
	Word embedding ‒ origins and fundamentals
	Distributed representations
	Static embeddings
		Word2Vec
		GloVe
	Creating your own embedding using gensim
	Exploring the embedding space with gensim
	Using word embeddings for spam detection
		Getting the data
		Making the data ready for use
		Building the embedding matrix
		Define the spam classifier
		Train and evaluate the model
		Running the spam detector
	Neural embeddings – not just for words
		Item2Vec
		node2vec
	Character and subword embeddings
	Dynamic embeddings
	Language model-based embeddings
		Using BERT as a feature extractor
		Fine-tuning BERT
		Classifying with BERT ‒ command line
		Using BERT as part of your own network
	Summary
	References
Chapter 8: Recurrent Neural Networks
	The basic RNN cell
		Backpropagation through time (BPTT)
		Vanishing and exploding gradients
	RNN cell variants
		Long short-term memory (LSTM)
		Gated recurrent unit (GRU)
		Peephole LSTM
	RNN variants
		Bidirectional RNNs
		Stateful RNNs
	RNN topologies
		Example ‒ One-to-Many – learning to generate text
		Example ‒ Many-to-One – Sentiment Analysis
		Example ‒ Many-to-Many – POS tagging
	Encoder-Decoder architecture – seq2seq
		Example ‒ seq2seq without attention for machine translation
	Attention mechanism
		Example ‒ seq2seq with attention for machine translation
	Transformer architecture
	Summary
	References
Chapter 9: Autoencoders
	Introduction to autoencoders
	Vanilla autoencoders
		TensorFlow Keras layers ‒ defining custom layers
		Reconstructing handwritten digits using an autoencoder
	Sparse autoencoder
	Denoising autoencoders
		Clearing images using a Denoising autoencoder
	Stacked autoencoder
		Convolutional autoencoder for removing noise from images
		Keras autoencoder example ‒ sentence vectors
	Summary
	References
Chapter 10: Unsupervised Learning
	Principal component analysis
		PCA on the MNIST dataset
		TensorFlow Embedding API
		K-means clustering
		K-means in TensorFlow 2.0
		Variations in k-means
	Self-organizing maps
		Colour mapping using SOM
	Restricted Boltzmann machines
		Reconstructing images using RBM
		Deep belief networks
	Variational Autoencoders
	Summary
	References
Chapter 11: Reinforcement Learning
	Introduction
		RL lingo
		Deep reinforcement learning algorithms
		Reinforcement success in recent years
	Introduction to OpenAI Gym
		Random agent playing Breakout
	Deep Q-Networks
		DQN for CartPole
		DQN to play a game of Atari
		DQN variants
			Double DQN
			Dueling DQN
			Rainbow
	Deep deterministic policy gradient
	Summary
	References
Chapter 12: TensorFlow and Cloud
	Deep learning on cloud
		Microsoft Azure
		Amazon Web Services (AWS)
		Google Cloud Platform (GCP)
		IBM Cloud
	Virtual machines on cloud
		EC2 on Amazon
		Compute Instance on GCP
		Virtual machine on Microsoft Azure
	Jupyter Notebooks on cloud
		SageMaker
		Google Colaboratory
		Microsoft Azure Notebooks
	TensorFlow Extended for production
		TFX Pipelines
		TFX pipeline components
		TFX libraries
	TensorFlow Enterprise
	Summary
	References
Chapter 13: TensorFlow for Mobile and IoT and TensorFlow.js
	TensorFlow Mobile
	TensorFlow Lite
		Quantization
		FlatBuffers
		Mobile converter
		Mobile optimized interpreter
		Supported platforms
		Architecture
		Using TensorFlow Lite
		A generic example of application
		Using GPUs and accelerators
		An example of application
	Pretrained models in TensorFlow Lite
		Image classification
		Object detection
		Pose estimation
		Smart reply
		Segmentation
		Style transfer
		Text classification
		Question and answering
		A note about using mobile GPUs
	An overview of federated learning at the edge
		TensorFlow FL APIs
	TensorFlow.js
		Vanilla TensorFlow.js
		Converting models
		Pretrained models
		Node.js
	Summary
	References
Chapter 14: An introduction to AutoML
	What is AutoML?
	Achieving AutoML
	Automatic data preparation
	Automatic feature engineering
	Automatic model generation
	AutoKeras
	Google Cloud AutoML
		Using Cloud AutoML ‒ Tables solution
		Using Cloud AutoML ‒ Vision solution
		Using Cloud AutoML ‒ Text Classification solution
		Using Cloud AutoML ‒ Translation solution
		Using Cloud AutoML ‒ Video Intelligence Classification solution
		Cost
	Bringing Google AutoML to Kaggle
	Summary
	References
Chapter 15: The Math Behind Deep Learning
	History
	Some mathematical tools
		Derivatives and gradients everywhere
		Gradient descent
		Chain rule
		A few differentiation rules
		Matrix operations
	Activation functions
		Derivative of the sigmoid
		Derivative of tanh
		Derivative of ReLU
	Backpropagation
		Forward step
		Backstep
			Case 1 – From hidden layer to output layer
			Case 2 ‒ From hidden layer to hidden layer
		Limit of backpropagation
		Cross entropy and its derivative
		Batch gradient descent, stochastic gradient descent, and mini-batch
			Batch Gradient Descent (BGD)
			Stochastic Gradient Descent (SGD)
			Mini-Batch Gradient Descent (MBGD)
	Thinking about backpropagation and convnets
	Thinking about backpropagation and RNNs
	A note on TensorFlow and automatic differentiation
	Summary
	References
Chapter 16: Tensor Processing Unit
	C/G/T processing units
		CPUs and GPUs
		TPUs
	Three generations of TPUs and Edge TPU
		First-generation TPU
		Second-generation TPU
		Third-generation TPU
		Edge TPU
	TPU performance
	How to use TPUs with Colab
		Checking whether TPUs are available
		Loading data with tf.data
		Building a model and loading it into the TPU
	Using pretrained TPU models
	Using TensorFlow 2.1 and nightly build
	Summary
	References
Other Books You May Enjoy
Index