Foundations of Deep Reinforcement Learning: Theory and Practice in Python

Cover
Title Page
Copyright Page
Contents
Foreword
Preface
Acknowledgments
About the Authors
1 Introduction to Reinforcement Learning
	1.1 Reinforcement Learning
	1.2 Reinforcement Learning as MDP
	1.3 Learnable Functions in Reinforcement Learning
	1.4 Deep Reinforcement Learning Algorithms
		1.4.1 Policy-Based Algorithms
		1.4.2 Value-Based Algorithms
		1.4.3 Model-Based Algorithms
		1.4.4 Combined Methods
		1.4.5 Algorithms Covered in This Book
		1.4.6 On-Policy and Off-Policy Algorithms
		1.4.7 Summary
	1.5 Deep Learning for Reinforcement Learning
	1.6 Reinforcement Learning and Supervised Learning
		1.6.1 Lack of an Oracle
		1.6.2 Sparsity of Feedback
		1.6.3 Data Generation
	1.7 Summary
Part I: Policy-Based and Value-Based Algorithms
	2 REINFORCE
		2.1 Policy
		2.2 The Objective Function
		2.3 The Policy Gradient
			2.3.1 Policy Gradient Derivation
		2.4 Monte Carlo Sampling
		2.5 REINFORCE Algorithm
			2.5.1 Improving REINFORCE
		2.6 Implementing REINFORCE
			2.6.1 A Minimal REINFORCE Implementation
			2.6.2 Constructing Policies with PyTorch
			2.6.3 Sampling Actions
			2.6.4 Calculating Policy Loss
			2.6.5 REINFORCE Training Loop
			2.6.6 On-Policy Replay Memory
		2.7 Training a REINFORCE Agent
		2.8 Experimental Results
			2.8.1 Experiment: The Effect of Discount Factor γ
			2.8.2 Experiment: The Effect of Baseline
		2.9 Summary
		2.10 Further Reading
		2.11 History
	3 SARSA
		3.1 The Q- and V-Functions
		3.2 Temporal Difference Learning
			3.2.1 Intuition for Temporal Difference Learning
		3.3 Action Selection in SARSA
			3.3.1 Exploration and Exploitation
		3.4 SARSA Algorithm
			3.4.1 On-Policy Algorithms
		3.5 Implementing SARSA
			3.5.1 Action Function: ɛ-Greedy
			3.5.2 Calculating the Q-Loss
			3.5.3 SARSA Training Loop
			3.5.4 On-Policy Batched Replay Memory
		3.6 Training a SARSA Agent
		3.7 Experimental Results
			3.7.1 Experiment: The Effect of Learning Rate
		3.8 Summary
		3.9 Further Reading
		3.10 History
	4 Deep Q-Networks (DQN)
		4.1 Learning the Q-Function in DQN
		4.2 Action Selection in DQN
			4.2.1 The Boltzmann Policy
		4.3 Experience Replay
		4.4 DQN Algorithm
		4.5 Implementing DQN
			4.5.1 Calculating the Q-Loss
			4.5.2 DQN Training Loop
			4.5.3 Replay Memory
		4.6 Training a DQN Agent
		4.7 Experimental Results
			4.7.1 Experiment: The Effect of Network Architecture
		4.8 Summary
		4.9 Further Reading
		4.10 History
	5 Improving DQN
		5.1 Target Networks
		5.2 Double DQN 106
		5.3 Prioritized Experience Replay (PER) 109
			5.3.1 Importance Sampling
		5.4 Modified DQN Implementation
			5.4.1 Network Initialization
			5.4.2 Calculating the Q-Loss
			5.4.3 Updating the Target Network
			5.4.4 DQN with Target Networks
			5.4.5 Double DQN
			5.4.6 Prioritized Experienced Replay
		5.5 Training a DQN Agent to Play Atari Games
		5.6 Experimental Results
			5.6.1 Experiment: The Effect of Double DQN and PER
		5.7 Summary
		5.8 Further Reading
Part II: Combined Methods
	6 Advantage Actor-Critic (A2C)
		6.1 The Actor
		6.2 The Critic
			6.2.1 The Advantage Function
			6.2.2 Learning the Advantage Function
		6.3 A2C Algorithm
		6.4 Implementing A2C
			6.4.1 Advantage Estimation
			6.4.2 Calculating Value Loss and Policy Loss
			6.4.3 Actor-Critic Training Loop
		6.5 Network Architecture
		6.6 Training an A2C Agent
			6.6.1 A2C with n-Step Returns on Pong
			6.6.2 A2C with GAE on Pong
			6.6.3 A2C with n-Step Returns on BipedalWalker
		6.7 Experimental Results
			6.7.1 Experiment: The Effect of n-Step Returns
			6.7.2 Experiment: The Effect of λ of GAE
		6.8 Summary
		6.9 Further Reading
		6.10 History
	7 Proximal Policy Optimization (PPO)
		7.1 Surrogate Objective
			7.1.1 Performance Collapse
			7.1.2 Modifying the Objective
		7.2 Proximal Policy Optimization (PPO)
		7.3 PPO Algorithm
		7.4 Implementing PPO
			7.4.1 Calculating the PPO Policy Loss
			7.4.2 PPO Training Loop
		7.5 Training a PPO Agent
			7.5.1 PPO on Pong
			7.5.2 PPO on BipedalWalker
		7.6 Experimental Results
			7.6.1 Experiment: The Effect of λ of GAE
			7.6.2 Experiment: The Effect of Clipping Variable ε
		7.7 Summary
		7.8 Further Reading
	8 Parallelization Methods
		8.1 Synchronous Parallelization
		8.2 Asynchronous Parallelization
			8.2.1 Hogwild!
		8.3 Training an A3C Agent
		8.4 Summary
		8.5 Further Reading
	9 Algorithm Summary
Part III: Practical Details
	10 Getting Deep RL to Work
		10.1 Software Engineering Practices
			10.1.1 Unit Tests
			10.1.2 Code Quality
			10.1.3 Git Workflow
		10.2 Debugging Tips
			10.2.1 Signs of Life
			10.2.2 Policy Gradient Diagnoses
			10.2.3 Data Diagnoses
			10.2.4 Preprocessor
			10.2.5 Memory
			10.2.6 Algorithmic Functions
			10.2.7 Neural Networks
			10.2.8 Algorithm Simplification
			10.2.9 Problem Simplification
			10.2.10 Hyperparameters
			10.2.11 Lab Workflow
		10.3 Atari Tricks
		10.4 Deep RL Almanac
			10.4.1 Hyperparameter Tables
			10.4.2 Algorithm Performance Comparison
		10.5 Summary
	11 SLM Lab
		11.1 Algorithms Implemented in SLM Lab
		11.2 Spec File
			11.2.1 Search Spec Syntax
		11.3 Running SLM Lab
			11.3.1 SLM Lab Commands
		11.4 Analyzing Experiment Results
			11.4.1 Overview of the Experiment Data
		11.5 Summary
	12 Network Architectures
		12.1 Types of Neural Networks
			12.1.1 Multilayer Perceptrons (MLPs)
			12.1.2 Convolutional Neural Networks (CNNs)
			12.1.3 Recurrent Neural Networks (RNNs)
		12.2 Guidelines for Choosing a Network Family
			12.2.1 MDPs vs. POMDPs
			12.2.2 Choosing Networks for Environments
		12.3 The Net API
			12.3.1 Input and Output Layer Shape Inference
			12.3.2 Automatic Network Construction
			12.3.3 Training Step
			12.3.4 Exposure of Underlying Methods
		12.4 Summary
		12.5 Further Reading
	13 Hardware
		13.1 Computer
		13.2 Data Types
		13.3 Optimizing Data Types in RL
		13.4 Choosing Hardware
		13.5 Summary
Part IV: Environment Design
	14 States
		14.1 Examples of States
		14.2 State Completeness
		14.3 State Complexity
		14.4 State Information Loss
			14.4.1 Image Grayscaling
			14.4.2 Discretization
			14.4.3 Hash Conflict
			14.4.4 Metainformation Loss
		14.5 Preprocessing
			14.5.1 Standardization
			14.5.2 Image Preprocessing
			14.5.3 Temporal Preprocessing
		14.6 Summary
	15 Actions
		15.1 Examples of Actions
		15.2 Action Completeness
		15.3 Action Complexity
		15.4 Summary
		15.5 Further Reading: Action Design in Everyday Things
	16 Rewards
		16.1 The Role of Rewards
		16.2 Reward Design Guidelines
		16.3 Summary
	17 Transition Function
		17.1 Feasibility Checks
		17.2 Reality Check
		17.3 Summary
Epilogue
A: Deep Reinforcement Learning Timeline
B: Example Environments
	B.1 Discrete Environments
		B.1.1 CartPole-v0
		B.1.2 MountainCar-v0
		B.1.3 LunarLander-v2
		B.1.4 PongNoFrameskip-v4
		B.1.5 BreakoutNoFrameskip-v4
	B.2 Continuous Environments
		B.2.1 Pendulum-v0
		B.2.2 BipedalWalker-v2
References
Index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X
	Z