The 2x2 Matrix: Contingency, Confusion and the Metrics of Binary Classification

Preface to the Second Edition
	References
Preface to the First Edition
	References
Contents
1 Introduction
	1.1 History and Nomenclature
	1.2 The Fourfold (2 × 2) Contingency Table
	1.3 Marginal Totals and Marginal Probabilities
		1.3.1 Marginal Totals
		1.3.2 Marginal Probabilities; P, Q
		1.3.3 Pre-test Odds
	1.4 Type I (α) and Type II (β) Errors
	1.5 Calibration: Decision Thresholds or Cut-Offs
	1.6 Uncertain or Inconclusive Test Results
	1.7 Measures Derived from a 2 × 2 Contingency Table; Confidence Intervals
	References
2 Paired Measures
	2.1 Introduction
	2.2 Error-Based Measures
		2.2.1 Sensitivity (Sens) and Specificity (Spec), or True Positive and True Negative Rates (TPR, TNR)
		2.2.2 False Positive Rate (FPR), False Negative Rate (FNR)
	2.3 Information-Based Measures
		2.3.1 Positive and Negative Predictive Values (PPV, NPV)
		2.3.2 False Discovery Rate (FDR), False Reassurance Rate (FRR)
		2.3.3 Bayes’ Formula; Standardized Positive and Negative Predictive Values (SPPV, SNPV)
		2.3.4 Interrelations of Sens, Spec, PPV, NPV, P, and Q
		2.3.5 Positive and Negative Likelihood Ratios (PLR, NLR)
		2.3.6 Post-test Odds; Net Harm to Net Benefit (H/B) Ratio
		2.3.7 Conditional Probability Plot
		2.3.8 Positive and Negative Predictive Ratios (PPR, NPR)
	2.4 Association-Based Measures
		2.4.1 Diagnostic Odds Ratio (DOR) and Error Odds Ratios (EOR)
		2.4.2 Positive and Negative Clinical Utility Indexes (PCUI, NCUI)
		2.4.3 Positive and Negative Clinical Disutility Indexes (PCDI, PCDI)
	References
3 Paired Complementary Measures
	3.1 Introduction
	3.2 Error-Based Measures
		3.2.1 Sensitivity (Sens) and False Negative Rate (FNR)
		3.2.2 Specificity (Spec) and False Positive Rate (FPR)
		3.2.3 “SnNout” and “SpPin” Rules
		3.2.4 Classification and Misclassification Rates; Misclassification Costs
		3.2.5 Accuracy (Acc) and Inaccuracy (Inacc)
		3.2.6 Balanced Accuracy and Inaccuracy (BAcc, BInacc)
		3.2.7 Unbiased Accuracy and Inaccuracy (UAcc, UInacc)
	3.3 Information-Based Measures
		3.3.1 Positive Predictive Value (PPV) and False Discovery Rate (FDR)
		3.3.2 Negative Predictive Value (NPV) and False Reassurance Rate (FRR)
		3.3.3 “Balanced Level” Formulations (BLAcc, BLInacc)
	3.4 Dependence of Paired Complementary Measures on Prevalence (P)
	References
4 Unitary Measures
	4.1 Introduction
	4.2 Youden Index (Y) or Statistic (J)
	4.3 Predictive Summary Index (PSI, Ψ)
	4.4 Harmonic Mean of Y and PSI (HMYPSI)
	4.5 Matthews’ Correlation Coefficient (MCC)
	4.6 Identification Indexes
		4.6.1 Identification Index (II)
		4.6.2 Balanced Identification Index (BII)
		4.6.3 Unbiased Identification Index (UII)
	4.7 Net Reclassification Improvement (NRI)
	4.8 Methods to Combine Sens and PPV
		4.8.1 Critical Success Index (CSI) or Threat Score (TS)
		4.8.2 Equitable Threat Score (ETS) or Gilbert Skill Score
		4.8.3 F Measure (F) or F1 Score (Dice Co-efficient)
		4.8.4 F*: CSI by Another Name
	4.9 Summary Utility Index (SUI) and Summary Disutility Index (SDI)
	4.10 “Diagnostic Yield”
	References
5 Number Needed (Reciprocal) Measures and Their Combinations as Likelihoods
	5.1 Introduction
	5.2 Number Needed to Diagnose (NND and NND*)
	5.3 Number Needed to Predict (NNP)
	5.4 Number Needed to Screen (NNS)
	5.5 Number Needed to Misdiagnose (NNM)
	5.6 Likelihood to Be Diagnosed or Misdiagnosed (LDM)
	5.7 Likelihood to Be Predicted or Misdiagnosed (LPM)
	5.8 Number Needed to Classify Correctly (NNCC)
	5.9 Number Needed to Misclassify (NNMC)
	5.10 Likelihood to Classify Correctly or Misclassify (LCM)
	5.11 Efficiency Index (EI)
		5.11.1 Balanced Efficiency Index (BEI)
		5.11.2 Balanced Level Efficiency Index (BLEI)
		5.11.3 Unbiased Efficiency Index (UEI)
	5.12 Number Needed for Screening Utility (NNSU)
	5.13 Number Needed for Screening Disutility (NNSD)
	5.14 Likelihood for Screening Utility or Disutility (LSUD)
	5.15 Comparing Likelihood Measures
	References
6 Quality (Q) Measures
	6.1 Introduction
	6.2 Error-Based Quality Measures
		6.2.1 Quality Sensitivity and Specificity (QSens, QSpec)
		6.2.2 Quality False Positive and False Negative Rates (QFPR, QFNR)
		6.2.3 Quality Accuracy, Inaccuracy (QAcc, QInacc)
		6.2.4 Quality Balanced Accuracy, Inaccuracy (QBAcc, QBInacc)
	6.3 Information-Based Quality Measures
		6.3.1 Quality Positive and Negative Predictive Values (QPPV, QNPV)
		6.3.2 Quality False Discovery and False Reassurance Rates (QFDR, QFRR)
		6.3.3 Quality Balanced Level Accuracy, Inaccuracy (QBLAcc, QBInacc)
		6.3.4 Quality Positive and Negative Likelihood Ratios (QPLR, QNLR)
		6.3.5 Quality Positive and Negative Predictive Ratios (QPPR, QNPR)
	6.4 Association-Based Quality Measures
		6.4.1 Quality Diagnostic Odds Ratio (QDOR)
		6.4.2 Quality Positive and Negative Clinical Utility Indexes (QPCUI, QNCUI)
		6.4.3 Quality Positive and Negative Clinical Disutility Indexes (QPCDI, QNCDI)
	6.5 Unitary Quality Measures
		6.5.1 Quality Youden Index (QY)
		6.5.2 Quality Predictive Summary Index (QPSI)
		6.5.3 Quality Harmonic Mean of Y and PSI (QHMYPSI)
		6.5.4 Quality Matthews’ Correlation Coefficient (QMCC)
		6.5.5 Quality Identification Index (QII)
		6.5.6 Quality Critical Success Index (QCSI) and Quality F Measure (QF)
		6.5.7 Quality Summary Utility Index and Summary Disutility Index (QSUI, QSDI)
	6.6 Quality Number Needed (Reciprocal) Measures and Their Combinations as Quality Likelihoods
		6.6.1 Quality Number Needed to Diagnose (QNND and QNND*)
		6.6.2 Quality Number Needed to Predict (QNNP)
		6.6.3 Quality Number Needed to Misdiagnose (QNNM)
		6.6.4 Quality Likelihood to Be Diagnosed or Misdiagnosed (QLDM)
		6.6.5 Quality Likelihood to Be Predicted or Misdiagnosed (QLPM)
		6.6.6 Quality Number Needed to Classify Correctly (QNNCC)
		6.6.7 Quality Number Needed to Misclassify (QNNMC)
		6.6.8 Quality Likelihood to Classify Correctly or Misclassify (QLCM)
		6.6.9 Quality Efficiency Index (QEI)
		6.6.10 Quality Balanced Efficiency Index (QBEI)
		6.6.11 Quality Balanced Level Efficiency Index (QBLEI)
		6.6.12 Quality Number Needed for Screening Utility (QNNSU)
		6.6.13 Quality Number Needed for Screening Disutility (QNNSD)
		6.6.14 Quality Likelihood for Screening Utility or Disutility (QLSUD)
	References
7 Graphing Methods
	7.1 Introduction
	7.2 Receiver Operating Characteristic (ROC) Plot or Curve
		7.2.1 The ROC Curve or Plot
		7.2.2 Area Under the ROC Curve (AUC)
	7.3 Defining Optimal Cut-Offs from the ROC Curve
		7.3.1 Youden Index (Y)
		7.3.2 Euclidean Index (D)
		7.3.3 Q* Index
		7.3.4 Other ROC-Based Methods
		7.3.5 Diagnostic Odds Ratio (DOR)
		7.3.6 Non-ROC-Based Methods
	7.4 Other Graphing Methods
		7.4.1 ROC Plot in Likelihood Ratio Coordinates
		7.4.2 Precision-Recall (PR) Plot or Curve
		7.4.3 Prevalence Value Accuracy Plots
		7.4.4 Agreement Charts
	References
8 Other Measures, Other Tables
	8.1 Introduction
	8.2 Combining Test Results
		8.2.1 Bayesian Method
		8.2.2 Boolean Method: “AND” and “OR” Logical Operators
		8.2.3 Decision Trees: “if-and-then” Rules
	8.3 Effect Sizes
		8.3.1 Correlation Coefficient
		8.3.2 Cohen’s d
		8.3.3 Binomial Effect Size Display (BESD)
	8.4 Other Measures of Association, Agreement, and Difference
		8.4.1 McNemar’s Test
		8.4.2 Cohen’s Kappa (κ) Statistic
		8.4.3 Bland–Altman Method
	8.5 Other Tables
		8.5.1 Higher Order Tables
		8.5.2 Interval Likelihood Ratios (ILRs)
		8.5.3 Three-Way Classification (Trichotomisation)
		8.5.4 Fourfold Pattern of Risk Attitudes
	References
9 Classification of Metrics of Binary Classification
	9.1 Introduction
	9.2 Classification of Metrics of Binary Classification
		9.2.1 Error/Information/Association-Based
		9.2.2 Descriptive Versus Predictive
		9.2.3 Statistical: Frequentist Versus Bayesian
		9.2.4 Test-oriented Versus Patient-Oriented
		9.2.5 Range
	9.3 Conclusion
		9.3.1 Fourfolds, Uncertainty, and the Epistemological Matrix
		9.3.2 Which Measure(s) Should Be Used?
	References
Index