Introduction to Probability for Data Science

Front Cover
Introduction to Probability for Data Science
Copyright
Dedication
Preface
Contents
1 Mathematical Background
	1.1 Infinite Series
		1.1.1 Geometric Series
		1.1.2 Binomial Series
	1.2 Approximation
		1.2.1 Taylor approximation
		1.2.2 Exponential series
		1.2.3 Logarithmic approximation
	1.3 Integration
		1.3.1 Odd and even functions
		1.3.2 Fundamental Theorem of Calculus
	1.4 Linear Algebra
		1.4.1 Why do we need linear algebra in data science?
		1.4.2 Everything you need to know about linear algebra
		1.4.3 Inner products and norms
		1.4.4 Matrix calculus
	1.5 Basic Combinatorics
		1.5.1 Birthday paradox
		1.5.2 Permutation
		1.5.3 Combination
	1.6 Summary
	1.7 Reference
	1.8 Problems
2 Probability
	2.1 Set Theory
		2.1.1 Why study set theory?
		2.1.2 Basic concepts of a set
		2.1.3 Subsets
		2.1.4 Empty set and universal set
		2.1.5 Union
		2.1.6 Intersection
		2.1.7 Complement and difference
		2.1.8 Disjoint and partition
		2.1.9 Set operations
		2.1.10 Closing remarks about set theory
	2.2 Probability Space
		2.2.1 Sample space Omega
		2.2.2 Event space F
		2.2.3 Probability law P
		2.2.4 Measure zero sets
		2.2.5 Summary of the probability space
	2.3 Axioms of Probability
		2.3.1 Why these three probability axioms?
		2.3.2 Axioms through the lens of measure
		2.3.3 Corollaries derived from the axioms
	2.4 Conditional Probability
		2.4.1 Definition of conditional probability
		2.4.2 Independence
		2.4.3 Bayes' theorem and the law of total probability
		2.4.4 The Three Prisoners problem
	2.5 Summary
	2.6 References
	2.7 Problems
3 Discrete Random Variables
	3.1 Random Variables
		3.1.1 A motivating example
		3.1.2 Definition of a random variable
		3.1.3 Probability measure on random variables
	3.2 Probability Mass Function
		3.2.1 Definition of probability mass function
		3.2.2 PMF and probability measure
		3.2.3 Normalization property
		3.2.4 PMF versus histogram
		3.2.5 Estimating histograms from real data
	3.3 Cumulative Distribution Functions (Discrete)
		3.3.1 Definition of the cumulative distribution function
		3.3.2 Properties of the CDF
		3.3.3 Converting between PMF and CDF
	3.4 Expectation
		3.4.1 Definition of expectation
		3.4.2 Existence of expectation
		3.4.3 Properties of expectation
		3.4.4 Moments and variance
	3.5 Common Discrete Random Variables
		3.5.1 Bernoulli random variable
		3.5.2 Binomial random variable
		3.5.3 Geometric random variable
		3.5.4 Poisson random variable
	3.6 Summary
	3.7 References
	3.8 Problems
4 Continuous Random Variables
	4.1 Probability Density Function
		4.1.1 Some intuitions about probability density functions
		4.1.2 More in-depth discussion about PDFs
		4.1.3 Connecting with the PMF
	4.2 Expectation, Moment, and Variance
		4.2.1 Definition and properties
		4.2.2 Existence of expectation
		4.2.3 Moment and variance
	4.3 Cumulative Distribution Function
		4.3.1 CDF for continuous random variables
		4.3.2 Properties of CDF
		4.3.3 Retrieving PDF from CDF
		4.3.4 CDF: Unifying discrete and continuous random variables
	4.4 Median, Mode, and Mean
		4.4.1 Median
		4.4.2 Mode
		4.4.3 Mean
	4.5 Uniform and Exponential Random Variables
		4.5.1 Uniform random variables
		4.5.2 Exponential random variables
		4.5.3 Origin of exponential random variables
		4.5.4 Applications of exponential random variables
	4.6 Gaussian Random Variables
		4.6.1 Definition of a Gaussian random variable
		4.6.2 Standard Gaussian
		4.6.3 Skewness and kurtosis
		4.6.4 Origin of Gaussian random variables
	4.7 Functions of Random Variables
		4.7.1 General principle
		4.7.2 Examples
	4.8 Generating Random Numbers
		4.8.1 General principle
		4.8.2 Examples
	4.9 Summary
	4.10 Reference
	4.11 Problems
5 Joint Distributions
	5.1 Joint PMF and Joint PDF
		5.1.1 Probability measure in 2D
		5.1.2 Discrete random variables
		5.1.3 Continuous random variables
		5.1.4 Normalization
		5.1.5 Marginal PMF and marginal PDF
		5.1.6 Independent random variables
		5.1.7 Joint CDF
	5.2 Joint Expectation
		5.2.1 Definition and interpretation
		5.2.2 Covariance and correlation coeffcient
		5.2.3 Independence and correlation
		5.2.4 Computing correlation from data
	5.3 Conditional PMF and PDF
		5.3.1 Conditional PMF
		5.3.2 Conditional PDF
	5.4 Conditional Expectation
		5.4.1 Definition
		5.4.2 The law of total expectation
	5.5 Sum of Two Random Variables
		5.5.1 Intuition through convolution
		5.5.2 Main result
		5.5.3 Sum of common distributions
	5.6 Random Vectors and Covariance Matrices
		5.6.1 PDF of random vectors
		5.6.2 Expectation of random vectors
		5.6.3 Covariance matrix
		5.6.4 Multidimensional Gaussian
	5.7 Transformation of Multidimensional Gaussians
		5.7.1 Linear transformation of mean and covariance
		5.7.2 Eigenvalues and eigenvectors
		5.7.3 Covariance matrices are always positive semi-definite
		5.7.4 Gaussian whitening
	5.8 Principal-Component Analysis
		5.8.1 The main idea: Eigendecomposition
		5.8.2 The eigenface problem
		5.8.3 What cannot be analyzed by PCA?
	5.9 Summary
	5.10 References
	5.11 Problems
6 Sample Statistics
	6.1 Moment-Generating and Characteristic Functions
		6.1.1 Moment-generating function
		6.1.2 Sum of independent variables via MGF
		6.1.3 Characteristic functions
	6.2 Probability Inequalities
		6.2.1 Union bound
		6.2.2 The Cauchy-Schwarz inequality
		6.2.3 Jensen's inequality
		6.2.4 Markov's inequality
		6.2.5 Chebyshev's inequality
		6.2.6 Chernoff's bound
		6.2.7 Comparing Chernoff and Chebyshev
		6.2.8 Hoeffding's inequality
	6.3 Law of Large Numbers
		6.3.1 Sample average
		6.3.2 Weak law of large numbers (WLLN)
		6.3.3 Convergence in probability
		6.3.4 Can we prove WLLN using Chernoff's bound?
		6.3.5 Does the weak law of large numbers always hold?
		6.3.6 Strong law of large numbers
		6.3.7 Almost sure convergence
		6.3.8 Proof of the strong law of large numbers
	6.4 Central Limit Theorem
		6.4.1 Convergence in distribution
		6.4.2 Central Limit Theorem
		6.4.3 Examples
		6.4.4 Limitation of the Central Limit Theorem
	6.5 Summary
	6.6 References
	6.7 Problems
7 Regression
	7.1 Principles of Regression
		7.1.1 Intuition: How to fit a straight line?
		7.1.2 Solving the linear regression problem
		7.1.3 Extension: Beyond a straight line
		7.1.4 Overdetermined and underdetermined systems
		7.1.5 Robust linear regression
	7.2 Overfitting
		7.2.1 Overview of overfitting
		7.2.2 Analysis of the linear case
		7.2.3 Interpreting the linear analysis results
	7.3 Bias and Variance Trade-Off
		7.3.1 Decomposing the testing error
		7.3.2 Analysis of the bias
		7.3.3 Variance
		7.3.4 Bias and variance on the learning curve
	7.4 Regularization
		7.4.1 Ridge regularization
		7.4.2 LASSO regularization
	7.5 Summary
	7.6 References
	7.7 Problems
8 Estimation
	8.1 Maximum-Likelihood Estimation
		8.1.1 Likelihood function
		8.1.2 Maximum-likelihood estimate
		8.1.3 Application 1: Social network analysis
		8.1.4 Application 2: Reconstructing images
		8.1.5 More examples of ML estimation
		8.1.6 Regression versus ML estimation
	8.2 Properties of ML Estimates
		8.2.1 Estimators
		8.2.2 Unbiased estimators
		8.2.3 Consistent estimators
		8.2.4 Invariance principle
	8.3 Maximum A Posteriori Estimation
		8.3.1 The trio of likelihood, prior, and posterior
		8.3.2 Understanding the priors
		8.3.3 MAP formulation and solution
		8.3.4 Analyzing the MAP solution
		8.3.5 Analysis of the posterior distribution
		8.3.6 Conjugate prior
		8.3.7 Linking MAP with regression
	8.4 Minimum Mean-Square Estimation
		8.4.1 Positioning the minimum mean-square estimation
		8.4.2 Mean squared error
		8.4.3 MMSE estimate = conditional expectation
		8.4.4 MMSE estimator for multidimensional Gaussian
		8.4.5 Linking MMSE and neural networks
	8.5 Summary
	8.6 References
	8.7 Problems
9 Confidence and Hypothesis
	9.1 Confidence Interval
		9.1.1 The randomness of an estimator
		9.1.2 Understanding confidence intervals
		9.1.3 Constructing a confidence interval
		9.1.4 Properties of the confidence interval
		9.1.5 Student's t-distribution
		9.1.6 Comparing Student's t-distribution and Gaussian
	9.2 Bootstrapping
		9.2.1 A brute force approach
		9.2.2 Bootstrapping
	9.3 Hypothesis Testing
		9.3.1 What is a hypothesis?
		9.3.2 Critical-value test
		9.3.3 p-value test
		9.3.4 Z-test and T-test
	9.4 Neyman-Pearson Test
		9.4.1 Null and alternative distributions
		9.4.2 Type 1 and type 2 errors
		9.4.3 Neyman-Pearson decision
	9.5 ROC and Precision-Recall Curve
		9.5.1 Receiver Operating Characteristic (ROC)
		9.5.2 Comparing ROC curves
		9.5.3 The ROC curve in practice
		9.5.4 The Precision-Recall (PR) curve
	9.6 Summary
	9.7 Reference
	9.8 Problems
10 Random Processes
	10.1 Basic Concepts
		10.1.1 Everything you need to know about a random process
		10.1.2 Statistical and temporal perspectives
	10.2 Mean and Correlation Functions
		10.2.1 Mean function
		10.2.2 Autocorrelation function
		10.2.3 Independent processes
	10.3 Wide-Sense Stationary Processes
		10.3.1 Definition of a WSS process
		10.3.2 Properties of RX(tau)
		10.3.3 Physical interpretation of RX(tau)
	10.4 Power Spectral Density
		10.4.1 Basic concepts
		10.4.2 Origin of the power spectral density
	10.5 WSS Process through LTI Systems
		10.5.1 Review of linear time-invariant systems
		10.5.2 Mean and autocorrelation through LTI Systems
		10.5.3 Power spectral density through LTI systems
		10.5.4 Cross-correlation through LTI Systems
	10.6 Optimal Linear Filter
		10.6.1 Discrete-time random processes
		10.6.2 Problem formulation
		10.6.3 Yule-Walker equation
		10.6.4 Linear prediction
		10.6.5 Wiener filter
	10.7 Summary
	10.8 Appendix
		10.8.1 The Mean-Square Ergodic Theorem
	10.9 References
	10.10 Problems
A Appendix
Back Cover