Explainable and Interpretable Reinforcement Learning for Robotics (Synthesis Lectures on Artificial Intelligence and Machine Learning)

Acknowledgements
Contents
About the Authors
1 Introduction
	1.1 Motivation
	1.2 Background
		1.2.1 Reinforcement Learning
		1.2.2 Reinforcement Learning Versus Supervised, Unsupervised, and Imitation Learning
		1.2.3 Explainable Artificial Intelligence (XAI) and Explainable Reinforcement Learning (XRL)
		1.2.4 Explainable Robotics (X-Robotics)
	1.3 Selection Criteria
2 Classification System
	[DELETE]
		2.1 Prior Surveys on XRL or X-Robotics
		2.2 Existing Classification Terminology for XRL or X-Robotics
		2.3 The Attributes of our Classification System
			2.3.1 Hard Attributes
			2.3.2 Soft Attributes (General)
			2.3.3 Soft Attributes (Robot-Specific)
3 Explainable Methods Organized by Category
	[DELETE]
	3.1 Decision Tree
		3.1.1 Single Decision Tree
		3.1.2 Single Altered Decision Tree
		3.1.3 Multiple or Combined Decision Trees
	3.2 Saliency Maps
		3.2.1 Post-Hoc Saliency Maps via Backpropagation
		3.2.2 Intrinsic Saliency Maps
		3.2.3 Post-Hoc Saliency Maps via Input Perturbation
	3.3 Counterfactuals/Counterexamples
		3.3.1 Counterfactual by Input Perturbation or Extra Information
		3.3.2 Counterfactual by Model Checking
	3.4 State Transformation
		3.4.1 Dimension Reduction
		3.4.2 Meaningful Representation Learning
	3.5 Observation Based Methods
		3.5.1 Observation Analysis: Frequency or Statistical Techniques for Policy Understanding
		3.5.2 Observation Analysis: Human Communicative Trajectories for Goal Understanding
		3.5.3 Observation Analysis: A/B Testing
		3.5.4 Training Data Observation Analysis
		3.5.5 Interrogative Observation Analysis
	3.6 Custom Domain Language
	3.7 Constrained Learning
	3.8 Constrained Execution
	3.9 Hierarchical
		3.9.1 Hierarchical Skills or Goals
		3.9.2 Primitive Generation
	3.10 Machine-to-Human Templates
		3.10.1 Model-to-Text or Policy-to-Text Templates
		3.10.2 Query-Based NLP Templates
	3.11 Model Reconciliation
		3.11.1 Certain Model Reconciliation
		3.11.2 Uncertain Model Reconciliation
	3.12 Causal Methods
	3.13 Reward Decomposition
		3.13.1 Standard Reward Decomposition Methods
		3.13.2 Model Uncertainty Reward Decomposition
	3.14 Visualizations
	3.15 Instruction Following
	3.16 Symbolic Methods
		3.16.1 Symbolic Transformation
		3.16.2 Symbolic Reward
		3.16.3 Symbolic Learning
	3.17 Legibility or Readability
4 Key Considerations and Resources
	[DELETE]
	4.1 General Discussion
	4.2 Limitations of Some Methods in this Survey
	4.3 Human-Robot Interaction Considerations
	4.4 Legibility and Readability
	4.5 AI Safety
	4.6 Environments
5 Opportunities, Challenges, and Future Directions
	[DELETE]
	5.1 Opportunities, Challenges, and Future Directions
	5.2 Conclusion
References
Index