Foundations of Deep Reinforcement Learning: Theory and Practice in Python

Title Page
Contents
Preface
Acknowledgements
About the Authors
Chapter 1. Introduction
	1.1 Reinforcement Learning
	1.2 Reinforcement Learning as MDP
	1.3 Learnable Functions in Reinforcement Learning
	1.4 Deep Reinforcement Learning Algorithms
	1.5 Deep Learning for Reinforcement Learning
	1.6 Reinforcement Learning and Supervised Learning
	1.7 Summary
Part I: Policy-based & Value-based Algorithms
	Chapter 2. Reinforce
		2.1 Policy
		2.2 The Objective Function
		2.3 The Policy Gradient
		2.4 Monte Carlo Sampling
		2.5 REINFORCE Algorithm
		2.6 Implementing REINFORCE
		2.7 Training a REINFORCE Agent
		2.8 Experimental Results
		2.9 Summary
		2.10 Further Reading
		2.11 History
	Chapter 3. SARSA
		3.1 The Q and V Functions
		3.2 Temporal Difference Learning
		3.3 Action Selection in SARSA
		3.4 SARSA Algorithm
		3.5 Implementing SARSA
		3.6 Training a SARSA Agent
		3.7 Experimental Results
		3.8 Summary
		3.9 Further Reading
		3.10 History
	Chapter 4. Deep Q-Networks (DQN)
		4.1 Learning the Q-function in DQN
		4.2 Action Selection in DQN
		4.3 Experience Replay
		4.4 DQN Algorithm
		4.5 Implementing DQN
		4.6 Training a DQN Agent
		4.7 Experimental Results
		4.8 Summary
		4.9 Further Reading
		4.10 History
	Chapter 5. Improving DQN
		5.1 Target Networks
		5.2 Double DQN
		5.3 Prioritized Experience Replay (PER)
		5.4 Modified DQN Implementation
		5.5 Training a DQN Agent to Play Atari Games
		5.6 Experimental Results
		5.7 Summary
		5.8 Further Reading
Part II: Combined methods
	Chapter 6. Advantage Actor-Critic (A2C)
		6.1 The Actor
		6.2 The Critic
		6.3 A2C Algorithm
		6.4 Implementing A2C
		6.5 Network Architecture
		6.6 Training an A2C Agent
		6.7 Experimental Results
		6.8 Summary
		6.9 Further Reading
		6.10 History
	Chapter 7. Proximal Policy Optimization (PPO)
		7.1 Surrogate Objective
		7.2 Proximal Policy Optimization (PPO)
		7.3 PPO Algorithm
		7.4 Implementing PPO
		7.5 Training a PPO Agent
		7.6 Experimental Results
		7.7 Summary
		7.8 Further Reading
	Chapter 8. Parallelization Methods
		8.1 Synchronous Parallelization
		8.2 Asynchronous Parallelization
		8.3 Training an A3C Agent
		8.4 Summary
		8.5 Further Reading
	Chapter 9. Algorithm Summary
Part III: Practical Tips
	Chapter 10. Getting Deep RL to Work
		10.1 Software Engineering Practices
		10.2 Debugging Tips
		10.3 Atari Tricks
		10.4 Deep RL Almanac
		10.5 Summary
	Chapter 11. SLM Lab
		11.1 Implemented Algorithms in SLM Lab
		11.2 Spec File
		11.3 Running SLM Lab
		11.4 Analyzing Experiment Results
		11.5 Summary
	Chapter 12. Network architectures
		12.1 Types of Neural Network
		12.2 Guidelines For Choosing a Network Family
		12.3 The Net API
		12.4 Summary
		12.5 Further Reading
	Chapter 13. Hardware
		13.1 Computer
		13.2 Information In Hardware
		13.3 Choosing Hardware
		13.4 Summary
	Chapter 14. Environment Design
		14.1 States
		14.2 Actions
		14.3 Rewards
		14.4 Transition Function
		14.5 Summary
		14.6 Further Reading: Action Design in Everyday Things
Epilogue
Appendix A. Deep Reinforcement Learning Timeline
Appendix B. Example Environments
	B.1 Discrete Environments
	B.2 Continuous Environments
Bibliography