Deep Reinforcement Learning: Fundamentals, Research and Applications

Foreword
Preface
Acknowledgements
Contents
Editors and Contributors
	About the Editors
	About the Authors
Acronyms
Mathematical Notation
	Fundamentals
	Deep Reinforcement Learning
Introduction
	Artificial Intelligence
	Machine Learning
	Deep Learning
	Reinforcement Learning
	Deep Reinforcement Learning
	TensorLayer
	References
Part I Fundamentals
	1 Introduction to Deep Learning
		1.1 Introduction
		1.2 Perceptron
			1.2.1 One Output
			1.2.2 Bias and Decision Boundary
			1.2.3 More Than One Output
		1.3 Multilayer Perceptron (MLP)
		1.4 Activation Functions
		1.5 Loss Functions
			1.5.1 Cross-Entropy Loss
			1.5.2 Lp Norm
			1.5.3 Mean Squared Error
			1.5.4 Mean Absolute Error
		1.6 Optimization
			1.6.1 Gradient Descent and Error Back-Propagation
			1.6.2 Stochastic Gradient Descent and Adaptive Learning Rate
			1.6.3 Hyper-Parameter Selection
				Cross-Validation
		1.7 Regularization
			1.7.1 Overfitting
			1.7.2 Weight Decay
			1.7.3 Dropout
			1.7.4 Batch Normalization
			1.7.5 Other Methods for Alleviating Overfitting
		1.8 Convolutional Neural Networks
		1.9 Recurrent Neural Networks
		1.10 Deep Learning Examples
			1.10.1 Tensor and Gradients
			1.10.2 Define a Model
			1.10.3 Customized Layers
			1.10.4 MLP: Image Classification on MNIST
			1.10.5 CNN: Image Classification on CIFAR10
			1.10.6 RNN and Seq2seq: Chatbot
		References
	2 Introduction to Reinforcement Learning
		2.1 Introduction
		2.2 Bandits
			2.2.1 Online Prediction and Online Learning
			2.2.2 Stochastic Multi-Armed Bandit
			2.2.3 Adversarial Multi-Armed Bandit
			2.2.4 Contextual Bandits
		2.3 Markov Decision Process
			2.3.1 Markov Process
			2.3.2 Markov Reward Process
			2.3.3 Markov Decision Process
			2.3.4 Bellman Equation and Optimality
				Bellman Equation
				Solutions of Bellman Equation
				Optimal Value Functions
				Bellman Optimality Equation
			2.3.5 Other Important Concepts
				Deterministic and Stochastic Policies
				Partially Observed Markov Decision Process
			2.3.6 Summary of Terminology in Reinforcement Learning
		2.4 Dynamic Programming
			2.4.1 Policy Iteration
			2.4.2 Value Iteration
			2.4.3 Other DPs: Asynchronous DP, Approximate DP, Real-Time DP
		2.5 Monte Carlo
			2.5.1 Monte Carlo Prediction
			2.5.2 Monte Carlo Control
			2.5.3 Incremental Monte Carlo
		2.6 Temporal Difference Learning
			2.6.1 TD Prediction
				TD(λ)
			2.6.2 Sarsa: On-Policy TD Control
				Convergence of Sarsa
			2.6.3 Q-Learning: Off-Policy TD Control
				Convergence of Q-Learning
		2.7 Policy Optimization
			2.7.1 Overview
				Recap of RL Skeleton
			2.7.2 Value-Based Optimization
				Value Function Approximation
				Gradient-Based Value Function Approximation
				Example: Deep Q-Network
			2.7.3 Policy-Based Optimization
				Gradient-Based Optimization
				Example: REINFORCE Algorithm
				Gradient-Free Optimization
				Example: Cross-Entropy (CE) Method
			2.7.4 Combination of Policy-Based and Value-Based Methods
				Compatible Function Approximation
				Other Methods
		References
	3 Taxonomy of Reinforcement Learning Algorithms
		3.1 Model-Based and Model-Free
		3.2 Value-Based and Policy-Based
		3.3 Monte Carlo and Temporal Difference
		3.4 On-Policy and Off-Policy
		References
	4 Deep Q-Networks
		4.1 Introduction
		4.2 Background
		4.3 Sarsa and Q-Learning
		4.4 Why Deep Learning: Value Function Approximation
		4.5 DQN
		4.6 Double DQN
		4.7 Dueling DQN
		4.8 Prioritized Experience Replay
		4.9 Other Improvements: Multi-Step Learning, Noisy Nets, and Distributional Reinforcement Learning
		4.10 DQN Examples
			4.10.1 Related Gym Environment
			4.10.2 DQN
			4.10.3 Double DQN
			4.10.4 Dueling DQN
			4.10.5 Prioritized Experience Replay
			4.10.6 Distributed DQN
		References
	5 Policy Gradient
		5.1 Introduction
		5.2 REINFORCE: Vanilla Policy Gradient
		5.3 Actor-Critic
		5.4 Generative Adversarial Networks and Actor-Critic
		5.5 Synchronous Advantage Actor-Critic (A2C)
		5.6 Asynchronous Advantage Actor-Critic (A3C)
		5.7 Trust Region Policy Optimization (TRPO)
			5.7.1 Natural Gradient
		5.8 Proximal Policy Optimization (PPO)
		5.9 Actor Critic Using Kronecker-Factored Trust Region (ACKTR)
		5.10 Policy Gradient Examples
			5.10.1 Related Gym Environments
				Discrete Action Space: Atari Pong Game and CartPole
				Pong
				CartPole
				Continuous Action Space: BipedalWalker-v2 and Pendulum-v0
				BipedalWalker-v2
				Pendulum-v0
			5.10.2 REINFORCE: Atari Pong Game and CartPole-v0
				Pong
				CartPole
			5.10.3 AC: CartPole-v0
			5.10.4 A3C: BipedalWalker-v2
			5.10.5 TRPO: Pendulum-v0
			5.10.6 PPO: Pendulum-v0
		References
	6 Combine Deep Q-Networks with Actor-Critic
		6.1 Introduction
		6.2 Deep Deterministic Policy Gradient (DDPG)
		6.3 Twin Delayed Deep Deterministic Policy Gradient (TD3)
		6.4 Soft Actor-Critic (SAC)
			6.4.1 Soft Policy Iteration
			6.4.2 SAC
		6.5 Examples
			6.5.1 Related Gym Environment
			6.5.2 DDPG: Pendulum-v0
			6.5.3 TD3: Pendulum-v0
			6.5.4 SAC: Pendulum-v0
		References
Part II Research
	7 Challenges of Reinforcement Learning
		7.1 Sample Efficiency
		7.2 Learning Stability
		7.3 Catastrophic Interference
		7.4 Exploration
		7.5 Meta-Learning and Representation Learning
		7.6 Multi-Agent Reinforcement Learning
		7.7 Sim to Real
		7.8 Large-Scale Reinforcement Learning
		7.9 Others
		References
	8 Imitation Learning
		8.1 Introduction
		8.2 Behavioral Cloning: Supervised Learning Approach
			8.2.1 Challenges of BC
			8.2.2 Dataset Aggregation
			8.2.3 Variational Dropout
			8.2.4 Other Methods in BC
		8.3 Inverse Reinforcement Learning Approach
			8.3.1 Challenges of IRL
			8.3.2 Generative Adversarial Approach
			8.3.3 Generative Adversarial Network Guided Cost Learning (GAN-GCL)
			8.3.4 Adversarial Inverse Reinforcement Learning (AIRL)
		8.4 Imitation Learning from Observation (IfO)
			8.4.1 Model-Based
			8.4.2 Model-Free
			8.4.3 Challenges of IfO
		8.5 Probabilistic Methods
		8.6 IL as Initialization for RL
		8.7 Other Approaches of Leveraging Demonstrations in RL
			8.7.1 Feeding Demonstrations into Replay Buffer
			8.7.2 Normalized Actor-Critic
			8.7.3 Reward Shaping with Demonstrations
		8.8 Summary
		References
	9 Integrating Learning and Planning
		9.1 Introduction
		9.2 Model-Based Method
		9.3 Integrated Architectures
		9.4 Simulation-Based Search
			9.4.1 Simple Monte Carlo Search
			9.4.2 Monte Carlo Tree Search
			9.4.3 TD Search
		References
	10 Hierarchical Reinforcement Learning
		10.1 Introduction
		10.2 Options Framework
			10.2.1 STRategic Attentive Writer (STRAW)
			10.2.2 The Option-Critic Architecture
		10.3 Feudal Reinforcement Learning
			10.3.1 FeUdal Networks (FuNs)
			10.3.2 Off-policy Correction
		10.4 Other Works
		References
	11 Multi-Agent Reinforcement Learning
		11.1 Introduction
		11.2 Optimization and Equilibrium
			11.2.1 Nash Equilibrium
				Pure Strategy Nash Equilibrium
				Mixed Strategy Nash Equilibrium
			11.2.2 Correlated Equilibrium
			11.2.3 Stackelberg Equilibrium
		11.3 Competition and Cooperation
			11.3.1 Cooperation
			11.3.2 Zero-Sum Game
			11.3.3 Simultaneous Competition
			11.3.4 Sequential Competition
		11.4 Game Theoretical Framework
		References
	12 Parallel Computing
		12.1 Introduction
		12.2 Synchronization and Asynchronization
		12.3 Parallel Communication and Networking
		12.4 Distributed Reinforcement Learning Algorithms
			12.4.1 Asynchronous Advantage Actor-Critic
			12.4.2 Hybrid GPU/CPU A3C
			12.4.3 Distributed Proximal Policy Optimization
			12.4.4 IMPALA and SEED
			12.4.5 Ape-X, Reactor, and R2D2
			12.4.6 Gorila
		12.5 Distributed Computing Architecture
		References
Part III Applications
	13 Learning to Run
		13.1 NIPS 2017 Challenge: Learning to Run
			13.1.1 Introduction of the Environment
			13.1.2 Installation
		13.2 Training an Agent to Run
			13.2.1 Parallel Training
			13.2.2 Tricks
			13.2.3 Learning Results
		References
	14 Robust Image Enhancement
		14.1 Image Enhancement
		14.2 Reinforcement Learning for Robust Processing
		References
	15 AlphaZero
		15.1 Introduction
		15.2 Combinatorial Games
		15.3 Monte Carlo Tree Search
		15.4 AlphaZero: A General Algorithm for Board Games
		References
	16 Robot Learning in Simulation
		16.1 Robotics Simulation
			16.1.1 Install CoppeliaSim and PyRep
			16.1.2 Git Clone Our Project
			16.1.3 Assemble the Robot
		16.2 Reinforcement Learning for Robotics Tasks
			16.2.1 Parallel Training
			16.2.2 Learning Performance
			16.2.3 Domain Randomization
			16.2.4 Robot Learning Benchmark
			16.2.5 Other Simulators
		References
	17 Arena Platform for Multi-Agent Reinforcement Learning
		17.1 Intallation
		17.2 Build Game with Arena
			17.2.1 Simple One-Player Game
			17.2.2 Simple Two-Player Game with Reward Scheme
			17.2.3 Advanced Settings
				Reward Scheme
				More Agent Prefabs
			17.2.4 Export Binary Game
		17.3 MARL Training
			17.3.1 Setup X-Server
			17.3.2 Run Training
				Continuous Action Space
				Discrete Action Space
			17.3.3 Visualization
		References
	18 Tricks of Implementation
		18.1 Overview: How to Apply Deep Reinforcement Learning?
		18.2 Implementation
		18.3 Training and Debugging
		References
Part IV Summary
	19 Algorithm Table
		References
	20 Algorithm Cheatsheet
		20.1 Deep Learning
			20.1.1 Stochastic Gradient Descent
			20.1.2 Adam Optimizer
		20.2 Reinforcement Learning
			20.2.1 Bandit
			20.2.2 Dynamic Programming (DP)
			20.2.3 Monte Carlo (MC)
		20.3 Deep Reinforcement Learning
		20.4 Advanced Deep Reinforcement Learning
			20.4.1 Imitation Learning
			20.4.2 Model-Based Reinforcement Learning
			20.4.3 Hierarchical Reinforcement Learning
			20.4.4 Multi-Agent Reinforcement Learning
			20.4.5 Parallel Computing