Introduction To Conformal Prediction With Python : A Short Guide For Quantifying Uncertainty Of Machine Learning Models

Summary
Preface
Who This Book Is For
Introduction to Conformal Prediction
	We need uncertainty quantification
	Uncertainty has many sources
	Distinguish good from bad predictions
	Other approaches don't have guaranteed coverage
	Conformal prediction fills the gap
Getting Started with Conformal Prediction in Python
	Installing the software
	Let's classify some beans
	First try: a naive approach
	Second try: conformal classification
	Getting started with MAPIE
Intuition Behind Conformal Prediction
	Conformal prediction is a recipe
	Understand parallels to out-of-sample evaluation
	How to interpret prediction regions and coverage
	Conformal prediction and supervised learning
Classification
	Back to the beans
	The naive method doesn't work
	The Score method is simple but not adaptive
	Use Adaptive Prediction Sets (APS) for conditional coverage
	Top-k method for fixed size sets
	Regularized APS (RAPS) for small sets
	Group-balanced conformal prediction
	Class-Conditional APS (CCAPS) for coverage by class
	Guide for choosing a conformal classification method
Regression and Quantile Regression
	Motivation
	Rent Index Data
	Conformalized Mean Regression
	Conformalized Quantile Regression (CQR)
A Glimpse Beyond Classification and Regression
	Quickly categorize conformal prediction by task and score
	Time Series Forecasting
	Multi-Label Classification
	Outlier Detection
	Probability Calibration
	And many more tasks
	How to stay up to date
Design Your Own Conformal Predictor
	Steps to build your own conformal predictor
	Finding the right non-conformity score
	Start with a heuristic notion of uncertainty
	A general recipe for 1D uncertaity heuristics
	Metrics for evaluating conformal predictors
Q & A
	How do I choose the calibration size?
	How do I make conformal prediction reproducible?
	How does alpha affect the size of the prediction regions?
	What happens if I choose a large \alpha for conformal classification?
	How to interpret empty prediction sets?
	Can I use the same data for calibration and model evaluation?
	What if I find errors in the book or want to provide feedback?
Acknowledgements
References