Practical Guide to Applied Conformal Prediction in Python

Cover
Title Page
Copyright and Credits
Foreword
Contributors
Table of Contents
Preface
Part 1: Introduction
Chapter 1: Introducing Conformal Prediction
	Technical requirements
	Introduction to conformal prediction
		Understanding conformity measures
	The origins of conformal prediction
		The future of conformal prediction
	How conformal prediction differs from traditional machine learning
	The p-value and its role in conformal prediction
	Summary
Chapter 2: Overview of Conformal Prediction
	Understanding uncertainty quantification
		Aleatoric uncertainty
		Epistemic uncertainty
	Different ways to quantify uncertainty
	Quantifying uncertainty using conformal prediction
	Summary
Part 2: Conformal Prediction Framework
Chapter 3: Fundamentals of Conformal Prediction
	Fundamentals of conformal prediction
		Definition and principles
	Basic components of a conformal predictor
		Types of nonconformity measures
	Summary
Chapter 4: Validity and Efficiency of Conformal Prediction
	The validity of probabilistic predictors
		Classifier calibration
	The efficiency of probabilistic predictors
	Summary
Chapter 5: Types of Conformal Predictors
	Understanding classical predictors
		Applying TCP for classification problems
		Applying TCP for regression problems
		Advantages
	Understanding inductive conformal predictors
	Choosing the right conformal predictor
		Transductive conformal predictors
		Inductive conformal predictors
	Summary
Part 3: Applications of Conformal Prediction
Chapter 6: Conformal Prediction for Classification
	Classifier calibration
		Understanding the concepts of classifier calibration
	Evaluating calibration performance
	Various approaches to classifier calibration
		Histogram binning
		Platt scaling
		Isotonic regression
	Conformal prediction for classifier calibration
		Venn-ABERS conformal prediction
		Comparing calibration methods
	Open source tools for conformal prediction in classification problems
		Nonconformist
	Summary
Chapter 7: Conformal Prediction for Regression
	Uncertainty quantification for regression problems
		Understanding the types and sources of uncertainty in regression modeling
		The concept of prediction intervals
		Why do we need prediction intervals?
		How is it different from a confidence interval?
	Conformal prediction for regression problems
	Building prediction intervals and predictive distributions using conformal prediction
	Mechanics of CQR
		Quantile regression
		CQR
		Jackknife+
		Jackknife regression
		Jackknife+ regression
		Conformal predictive distributions
	Summary
Chapter 8: Conformal Prediction for Time Series and Forecasting
	UQ for time series and forecasting problems
		The importance of UQ
		The history of UQ
		Early statistical methods – the roots of UQ in time series
		Modern machine learning approaches
	The concept of PIs in forecasting applications
		Definition and construction
		The importance of forecasting applications
		Challenges and considerations
	Various approaches to producing PIs
		Parametric approaches
		Non-parametric approaches
		Bayesian approaches
		Machine learning approaches
		Conformal prediction
	Conformal prediction for time series and forecasting
		Ensemble batch PIs (EnbPIs)
		NeuralProphet
	Summary
Chapter 9: Conformal Prediction for Computer Vision
	Uncertainty quantification for computer vision
		Why does uncertainty matter?
		Types of uncertainty in computer vision
		Quantifying uncertainty
	Why does deep learning produce miscalibrated predictions?
		Post-2012 – the deep learning surge
		The "calibration crisis" in deep learning – a turning point in 2017
		Overconfidence in modern deep learning computer vision models
	Various approaches to quantify uncertainty in computer vision problems
	The superiority of conformal prediction in uncertainty quantification
	Conformal prediction for computer vision
		Uncertainty sets for image classifiers using conformal prediction
	Building computer vision classifiers using conformal prediction
		Naïve Conformal prediction
		Adaptive Prediction Sets (APS)
		Regularized Adaptive Prediction Sets (RAPS)
	Summary
Chapter 10: Conformal Prediction for Natural Language Processing
	Uncertainty quantification for NLP
		What is uncertainty in NLP?
		Benefits of quantifying uncertainty in NLP
		The challenges of uncertainty in NLP
	Understanding why deep learning produces miscalibrated predictions
		Introduction to deep learning in NLP
		Challenges with deep learning predictions in NLP
		The implications of miscalibration
	Various approaches to quantify uncertainty in NLP problems
		Bootstrap methods and ensemble techniques
		Out-of-distribution (OOD) detection
	Conformal prediction for NLP
		How conformal prediction works in NLP
		Practical applications of conformal prediction in NLP
		Advantages of using conformal prediction in NLP
	Summary
Part 4: Advanced Topics
Chapter 11: Handling Imbalanced Data
	Introducing imbalanced data
	Why imbalanced data problems are complex to solve
	Methods for solving imbalanced data
	The methods for solving imbalanced data
	Solving imbalanced data problems by applying conformal prediction
		Addressing imbalanced data with Venn-Abers predictors
		Key insights from the Credit Card Fraud Detection notebook
	Summary
Chapter 12: Multi-Class Conformal Prediction
	Multi-class classification problems
		Algorithms for multi-class classification
		One-vs-all and one-vs-one strategies
	Metrics for multi-class classification problems
		Confusion matrix
		Precision
		Recall
		F1 score
		Macro- and micro-averaged metrics
		Area Under Curve (AUC-ROC)
		Log loss and its application in measuring calibration of multi-class models
		Brier score and its application in measuring the calibration of multi-class models
	How conformal prediction can be applied to multi-class classification problems
		Multi-class probabilistic classification using inductive and cross-Venn-ABERS predictors
	Summary
Index
Other Books You May Enjoy