Deep Reinforcement Learning in Action

brief contents
contents
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized: A roadmap
	About the code
	liveBook discussion forum
about the authors
about the cover illustration
Part 1—Foundations
	1 What is reinforcement learning?
		1.1 The “deep” in deep reinforcement learning
		1.2 Reinforcement learning
		1.3 Dynamic programming versus Monte Carlo
		1.4 The reinforcement learning framework
		1.5 What can I do with reinforcement learning?
		1.6 Why deep reinforcement learning?
		1.7 Our didactic tool: String diagrams
		1.8 What’s next?
		Summary
	2 Modeling reinforcement learning problems: Markov decision processes
		2.1 String diagrams and our teaching methods
		2.2 Solving the multi-arm bandit
			2.2.1 Exploration and exploitation
			2.2.2 Epsilon-greedy strategy
			2.2.3 Softmax selection policy
		2.3 Applying bandits to optimize ad placements
			2.3.1 Contextual bandits
			2.3.2 States, actions, rewards
		2.4 Building networks with PyTorch
			2.4.1 Automatic differentiation
			2.4.2 Building Models
		2.5 Solving contextual bandits
		2.6 The Markov property
		2.7 Predicting future rewards: Value and policy functions
			2.7.1 Policy functions
			2.7.2 Optimal policy
			2.7.3 Value functions
		Summary
	3 Predicting the best states and actions: Deep Q-networks
		3.1 The Q function
		3.2 Navigating with Q-learning
			3.2.1 What is Q-learning?
			3.2.2 Tackling Gridworld
			3.2.3 Hyperparameters
			3.2.4 Discount factor
			3.2.5 Building the network
			3.2.6 Introducing the Gridworld game engine
			3.2.7 A neural network as the Q function
		3.3 Preventing catastrophic forgetting: Experience replay
			3.3.1 Catastrophic forgetting
			3.3.2 Experience replay
		3.4 Improving stability with a target network
			3.4.1 Learning instability
		3.5 Review
		Summary
	4 Learning to pick the best policy: Policy gradient methods
		4.1 Policy function using neural networks
			4.1.1 Neural network as the policy function
			4.1.2 Stochastic policy gradient
			4.1.3 Exploration
		4.2 Reinforcing good actions: The policy gradient algorithm
			4.2.1 Defining an objective
			4.2.2 Action reinforcement
			4.2.3 Log probability
			4.2.4 Credit assignment
		4.3 Working with OpenAI Gym
			4.3.1 CartPole
			4.3.2 The OpenAI Gym API
		4.4 The REINFORCE algorithm
			4.4.1 Creating the policy network
			4.4.2 Having the agent interact with the environment
			4.4.3 Training the model
			4.4.4 The full training loop
			4.4.5 Chapter conclusion
		Summary
	5 Tackling more complex problems with actor-critic methods
		5.1 Combining the value and policy function
		5.2 Distributed training
		5.3 Advantage actor-critic
		5.4 N-step actor-critic
		Summary
Part 2—Above and beyond
	6 Alternative optimization methods: Evolutionary algorithms
		6.1 A different approach to reinforcement learning
		6.2 Reinforcement learning with evolution strategies
			6.2.1 Evolution in theory
			6.2.2 Evolution in practice
		6.3 A genetic algorithm for CartPole
		6.4 Pros and cons of evolutionary algorithms
			6.4.1 Evolutionary algorithms explore more
			6.4.2 Evolutionary algorithms are incredibly sample intensive
			6.4.3 Simulators
		6.5 Evolutionary algorithms as a scalable alternative
			6.5.1 Scaling evolutionary algorithms
			6.5.2 Parallel vs. serial processing
			6.5.3 Scaling efficiency
			6.5.4 Communicating between nodes
			6.5.5 Scaling linearly
			6.5.6 Scaling gradient-based approaches
		Summary
	7 Distributional DQN: Getting the full story
		7.1 What’s wrong with Q-learning?
		7.2 Probability and statistics revisited
			7.2.1 Priors and posteriors
			7.2.2 Expectation and variance
		7.3 The Bellman equation
			7.3.1 The distributional Bellman equation
		7.4 Distributional Q-learning
			7.4.1 Representing a probability distribution in Python
			7.4.2 Implementing the Dist-DQN
		7.5 Comparing probability distributions
		7.6 Dist-DQN on simulated data
		7.7 Using distributional Q-learning to play Freeway
		Summary
	8 Curiosity-driven exploration
		8.1 Tackling sparse rewards with predictive coding
		8.2 Inverse dynamics prediction
		8.3 Setting up Super Mario Bros.
		8.4 Preprocessing and the Q-network
		8.5 Setting up the Q-network and policy function
		8.6 Intrinsic curiosity module
		8.7 Alternative intrinsic reward mechanisms
		Summary
	9 Multi-agent reinforcement learning
		9.1 From one to many agents
		9.2 Neighborhood Q-learning
		9.3 The 1D Ising model
		9.4 Mean field Q-learning and the 2D Ising model
		9.5 Mixed cooperative-competitive games
		Summary
	10 Interpretable reinforcement learning: Attention and relational models
		10.1 Machine learning interpretability with attention and relational biases
			10.1.1 Invariance and equivariance
		10.2 Relational reasoning with attention
			10.2.1 Attention models
			10.2.2 Relational reasoning
			10.2.3 Self-attention models
		10.3 Implementing self-attention for MNIST
			10.3.1 Transformed MNIST
			10.3.2 The relational module
			10.3.3 Tensor contractions and Einstein notation
			10.3.4 Training the relational module
		10.4 Multi-head attention and relational DQN
		10.5 Double Q-learning
		10.6 Training and attention visualization
			10.6.1 Maximum entropy learning
			10.6.2 Curriculum learning
			10.6.3 Visualizing attention weights
		Summary
	11 In conclusion: A review and roadmap
		11.1 What did we learn?
		11.2 The uncharted topics in deep reinforcement learning
			11.2.1 Prioritized experience replay
			11.2.2 Proximal policy optimization (PPO)
			11.2.3 Hierarchical reinforcement learning and the options framework
			11.2.4 Model-based planning
			11.2.5 Monte Carlo tree search (MCTS)
		11.3 The end
Appendix—Mathematics, deep learning, PyTorch
	A.1 Linear algebra
	A.2 Calculus
	A.3 Deep learning
	A.4 PyTorch
Reference list
index
	Symbols
	Numerics
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X
	Y