Integral and Inverse Reinforcement Learning for Optimal Control Systems and Games (Advances in Industrial Control)

Series Editor’s Foreword
Preface
Acknowledgements
Contents
Abbreviations and Notation
	Abbreviations
	Notation
1 Introduction
	1.1 Motivation
	1.2 Optimal Control
	1.3 Integral Reinforcement Learning
	1.4 Inverse Optimal Control
	1.5 Inverse Reinforcement Learning
	1.6 Outline of This Book
	References
2 Background on Integral and Inverse Reinforcement Learning for Feedback Control
	2.1 Integral Reinforcement Learning for Continuous-Time Systems
		2.1.1 Linear Quadratic Regulators
		2.1.2 Integral Reinforcement Learning
	2.2 Inverse Optimal Control for Continuous-Time Systems
		2.2.1 Inverse Optimal Control for Linear Systems
		2.2.2 Inverse Optimal Control for Nonlinear Systems
	References
Part I Integral Reinforcement Learning for Optimal Control Systems and Games
3 Integral Reinforcement Learning  for Optimal Regulation
	3.1 Introduction
	3.2 On-Policy Synchronous Integral Reinforcement Learning with Experience Replay for Nonlinear Constrained Systems
		3.2.1 Problem Formulation
		3.2.2 Offline Integral Reinforcement Learning Policy Iteration
		3.2.3 Value Function Approximation
		3.2.4 Synchronous Online Integral Reinforcement Learning for Nonlinear Constrained Systems
		3.2.5 Simulation Examples
	3.3 Off-Policy Integral Reinforcement Learning for Linear Quadratic Regulators with Input–Output Data
		3.3.1 Discounted Optimal Control Problem
		3.3.2 State-Feedback Off-Policy RL with Input-State Data
		3.3.3 Output-Feedback Off-Policy RL with Input–Output Data
		3.3.4 Simulation Examples
	References
4 Integral Reinforcement Learning for Optimal Tracking
	4.1 Introduction
	4.2 Integral Reinforcement Learning Policy Iteration for Linear …
		4.2.1 Problem Formulation
		4.2.2 Augmented Algebraic Riccati Equation for Causal Solution
		4.2.3 Integral Reinforcement Learning for Online Linear Quadratic Tracking
		4.2.4 Simulation Examples
	4.3 Online Actor–Critic Integral Reinforcement Learning …
		4.3.1 Standard Problem Formulation and Solution
		4.3.2 New Formulation for the Optimal Tracking Control Problem of Constrained-Input Systems
		4.3.3 Tracking Bellman and Hamilton–Jacobi–Bellman Equations
		4.3.4 Offline Policy Iteration Algorithms
		4.3.5 Online Actor–Critic-Based Integral Reinforcement Learning
	4.4 Simulation Results
	References
5 Integral Reinforcement Learning for Zero-Sum Games
	5.1 Introduction
	5.2 Off-Policy Integral Reinforcement Learning for upper H Subscript normal infinityHinfty Tracking Control
		5.2.1 Problem Formulation
		5.2.2 Tracking Hamilton–Jacobi–Isaacs Equation and the Solution Stability
		5.2.3 Off-Policy Integral Reinforcement Learning for Tracking Hamilton–Jacobi–Isaacs Equation
		5.2.4 Simulation Examples
	5.3 Off-Policy Integral Reinforcement Learning for Distributed …
		5.3.1 Formulation of Distributed Minmax Strategy
		5.3.2 Stability and Robustness of Distributed Minmax Strategy
		5.3.3 Off-Policy Integral Reinforcement Learning for Distributed Minmax Strategy
		5.3.4 Simulation Examples
	References
Part II Inverse Reinforcement Learning for Optimal Control Systems and Games
6 Inverse Reinforcement Learning for Optimal Control Systems
	6.1 Introduction
	6.2 Off-Policy Inverse Reinforcement Learning for Linear Quadratic Regulators
		6.2.1 Problem Formulation
		6.2.2 Inverse Reinforcement Learning Policy Iteration
		6.2.3 Model-Free Off-Policy Inverse Reinforcement Learning
		6.2.4 Simulation Examples
	6.3 Off-Policy Inverse Reinforcement Learning for Nonlinear …
		6.3.1 Problem Formulation
		6.3.2 Model-Based Inverse Reinforcement Learning
		6.3.3 Model-Free Off-Policy Integral Inverse Reinforcement Learning
		6.3.4 Simulation Examples
	References
7 Inverse Reinforcement Learning for Two-Player Zero-Sum 
Games
	7.1 Introduction
	7.2 Inverse Q-Learning for Linear Two-Player Zero-Sum Games
		7.2.1 Problem Formulation
		7.2.2 Model-Free Inverse Q-Learning
		7.2.3 Implementation of Inverse Q-Learning Algorithm
		7.2.4 Simulation Examples
	7.3 Off-Policy Inverse Reinforcement Learning for Nonlinear Two-Player Zero-Sum Games
		7.3.1 Problem Formulation
		7.3.2 Inverse Reinforcement Learning Policy Iteration
		7.3.3 Model-Free Off-Policy Integral Inverse Reinforcement 
Learning
		7.3.4 Simulation Examples
	7.4 Online Adaptive Inverse Reinforcement Learning for Nonlinear Two-Player Zero-Sum Games
		7.4.1 Integral RL-Based Off line Inverse Reinforcement Learning
		7.4.2 Online Inverse Reinforcement  Learning with Synchronous Neural Networks
		7.4.3 Simulation Examples
	References
8 Inverse Reinforcement Learning for Multiplayer Non-Zero-Sum Games
	8.1 Introduction
	8.2 Off-Policy Inverse Reinforcement Learning for Linear Multiplayer Non-Zero-Sum Games
		8.2.1 Problem Formulation
		8.2.2 Inverse Reinforcement Learning Policy Iteration
		8.2.3 Model-Free Off-Policy Integral Inverse  Reinforcement Learning
		8.2.4 Simulation Examples
	8.3 Off-Policy Inverse Reinforcement Learning for Nonlinear Multiplayer Non-Zero-Sum Games
		8.3.1 Problem Formulation
		8.3.2 Inverse Reinforcement Learning Policy Iteration
		8.3.3 Model-Free Off-Policy Integral Inverse  Reinforcement Learning
		8.3.4 Simulation Examples
	References
Appendix A Some Useful Facts in Matrix Algebra
Index