Machine Learning for High-Risk Applications

Preface
	Who Should Read This Book
	What Readers Will Learn
		Preliminary Book Outline
		Bringing it All Together
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
1. Contemporary Model Governance
	Basic Legal Obligations
	AI Incidents
	Organizational and Cultural Competencies for Responsible AI
		Accountability
		Drinking Your Own Champagne
		Diverse and Experienced Teams
		“Going Fast and Breaking Things”
	Organizational Processes for Responsible AI
		Forecasting Failure Modes
		Model Risk Management
		Beyond Model Risk Management
	Case Study: Death by Autonomous Vehicle
		Fallout
		An Unprepared Legal System
		Lessons Learned
2. How to Red-Team AI Systems
	Security Basics
		The Adversarial Mindset
		CIA Triad
		Best Practices for Data Scientists
	Machine Learning Attacks
		Integrity Attacks: Manipulated Machine Learning Outputs
		Confidentiality Attacks: Extracted Information
	General AI Security Concerns
	Counter-measures
		Model Debugging for Security
		Model Monitoring For Security
		Privacy-enhancing Technologies
		Robust Machine Learning
		General Countermeasures
	Case Study: Real-world Evasion Attacks
		Lessons Learned
	Resources
3. Debugging AI Systems for Safety and Performance
	Training
		Reproducibility
		Data Quality and Feature Engineering
		Model Specification
	Model Debugging
		Software Testing
		Traditional Model Assessment
		Residual Analysis for Machine Learning
		Sensitivity Analysis
		Benchmark Models
		Machine Learning Bugs
		Remediation: Fixing Bugs
	Deployment
		Domain Safety
		Model Monitoring
	Case Study: Remediating the Strawman
	Resources