Stochastic Methods for Modeling and Predicting Complex Dynamical Systems: Uncertainty Quantification, State Estimation, and Reduced-Order Models

Preface
Acknowledgements
Contents
1 Stochastic Toolkits
	1.1 Review of Basic Probability Concepts
		1.1.1 Random Variable, Probability Measure, and Probability Density Function (PDF)
		1.1.2 Gaussian Distribution
		1.1.3 Moments and Non-Gaussian Distributions
		1.1.4 Law of Large Numbers and Central Limit Theorem
	1.2 Stochastic Processes
	1.3 Stochastic Differential Equations (SDEs)
		1.3.1 Itô Stochastic Integral
		1.3.2 Itô's Formula
		1.3.3 Fokker-Planck Equation
	1.4 Markov Jump Processes
		1.4.1 A Motivating Example of Intermittent Time Series
		1.4.2 Finite-State Markov Jump Process
		1.4.3 Master Equation
		1.4.4 Switching Times
	1.5 Chaotic Systems
2 Introduction to Information Theory
	2.1 Shannon's Entropy and Maximum Entropy Principle
		2.1.1 Shannon's Intuition from the Theory of Communication
		2.1.2 Definition of Shannon's Entropy
		2.1.3 Shannon's Entropy in Gaussian Framework
		2.1.4 Maximum Entropy Principle
		2.1.5 Coarse Graining and the Loss of Information
	2.2 Relative Entropy, Quantifying the Model Error and Additional …
		2.2.1 Definition of Relative Entropy
		2.2.2 Relative Entropy in Gaussian Framework
		2.2.3 Maximum Relative Entropy Principle
	2.3 Mutual Information
		2.3.1 Definition of Mutual Information
		2.3.2 Mutual Information in Gaussian Framework
	2.4 Relationship Between Information and Path-Wise Measurements
		2.4.1 Two Widely Used Path-Wise Measurements: Root-Mean-Square Error (RMSE) and Pattern Correction
		2.4.2 Relationship Between Shannon's Entropy of Residual and RMSE
		2.4.3 Relationship Between Mutual Information and Pattern Correlation
		2.4.4 Why Relative Entropy is Important?
3 Basic Stochastic Computational Methods
	3.1 Monte Carlo Method
		3.1.1 The Basic Idea
		3.1.2 A Simple Example
		3.1.3 Error Estimates for the Monte Carlo Method
	3.2 Numerical Schemes for Solving SDEs
		3.2.1 Euler-Maruyama Scheme
		3.2.2 Milstein Scheme
	3.3 Ensemble Method for Solving the Statistics of SDEs
	3.4 Kernel Density Estimation
4 Simple Gaussian and Non-Gaussian SDEs
	4.1 Linear Gaussian SDEs
		4.1.1 Reynolds Decomposition and Time Evolution of the Moments
		4.1.2 Statistical Equilibrium State and Decorrelation Time
		4.1.3 Fokker-Planck Equation
	4.2 Non-Gaussian SDE: Linear Model with Multiplicative Noise
		4.2.1 Solving the Exact Path-Wise Solution
		4.2.2 Equilibrium Distribution
		4.2.3 Time Evolution of the Moments
	4.3 Non-Gaussian SDE: Scalar Model with Cubic Nonlinearity and Correlated Additive and Multiplicative (CAM) Noise
		4.3.1 Equilibrium PDF
		4.3.2 Time Evolution of the Moments
		4.3.3 Quasi-Gaussian Closure for the Moment Equations Associated with the Cubic Model
	4.4 Nonlinear SDEs with Exactly Solvable Conditional Moments
		4.4.1 The Coupled Nonlinear SDE System
		4.4.2 Derivation of the Exact Solvable Conditional Moments
5 Data Assimilation
	5.1 Introduction
	5.2 Kalman Filter
		5.2.1 One-Dimensional Kalman Filter: Basic Idea of Data Assimilation
		5.2.2 A Simple Example
		5.2.3 Multi-Dimensional Case
		5.2.4 Some Remarks
	5.3 Nonlinear Filters
		5.3.1 Extended Kalman Filter
		5.3.2 Ensemble Kalman Filter
		5.3.3 A Numerical Example
		5.3.4 Particle Filter
	5.4 Continuous-In-Time Version: Kalman-Bucy Filter
	5.5 Other Data Assimilation Methods and Applications
6 Prediction
	6.1 Ensemble Forecast
		6.1.1 Trajectory Forecast Versus Ensemble Forecast
		6.1.2 Lead Time and Ensemble Mean Forecast Time Series
	6.2 Model Error, Internal Predictability and Prediction Skill
		6.2.1 Important Factors for Useful Prediction
		6.2.2 Quantifying the Predictability and Model Error via Information Criteria
	6.3 Procedure of Designing Suitable Forecast Models
	6.4 Predicting Model Response via Fluctuation-Dissipation Theorem
		6.4.1 The FDT Framework
		6.4.2 Approximate FDT Methods
		6.4.3 A Nonlinear Example
	6.5 Finding the Most Sensitive Change Directions via Information Theory
		6.5.1 The Mathematical Framework Using Fisher Information
		6.5.2 Examples
		6.5.3 Practical Strategies Utilizing FDT
7 Data-Driven Low-Order Stochastic Models
	7.1 Motivations
	7.2 Complex Ornstein–Uhlenbeck (OU) Process
		7.2.1 Calibration of the OU Process
		7.2.2 Compensating the Effect of Complicated Nonlinearity by Simple Stochastic Noise
		7.2.3 Application: Reproducing the Statistics of the Two-Layer Quasi-Geostrophic Turbulence
	7.3 Combining Stochastic Models with Linear Analysis in PDEs to Model …
	7.4 Linear Stochastic Model with Multiplicative Noise
		7.4.1 Exact Formula for Model Calibration
		7.4.2 Approximating Highly Nonlinear Time Series
		7.4.3 Application: Characterizing an El Niño Index
	7.5 Stochastically Parameterized Models
		7.5.1 The Necessity of Stochastic Parameterization
		7.5.2 The Stochastic Parameterized Extended Kalman Filter (SPEKF) Model
		7.5.3 Filtering Intermittent Time Series
	7.6 Physics-Constrained Nonlinear Regression Models
		7.6.1 Motivations
		7.6.2 The General Framework of Physics-Constrained Nonlinear Regression Models
		7.6.3 Comparison of the Models with and Without Physics Constraints
	7.7 Discussion: Linear and Gaussian Approximations for Nonlinear Systems
8 Conditional Gaussian Nonlinear Systems
	8.1 Overview of the Conditional Gaussian Nonlinear System (CGNS)
		8.1.1 The Mathematical Framework
		8.1.2 Examples of Complex Dynamical Systems Belonging to the CGNS
		8.1.3 The Mathematical and Physical Reasonings Behind the CGNS
	8.2 Closed Analytic Formulae for Solving the Conditional Statistics
		8.2.1 Nonlinear Optimal Filtering
		8.2.2 Nonlinear Optimal Smoothing
		8.2.3 Nonlinear Optimal Conditional Sampling
		8.2.4 Example: Comparison of Filtering, Smoothing and Sampling
	8.3 Lagrangian Data Assimilation
		8.3.1 The Mathematical Setup
		8.3.2 Filtering Compressible and Incompressible Flows
		8.3.3 Uncertainty Quantification Using Information Theory
	8.4 Solving High-Dimensional Fokker-Planck Equations
		8.4.1 The Basic Algorithm
		8.4.2 A Simple Example
		8.4.3 Block Decomposition
		8.4.4 Statistical Symmetry
		8.4.5 Application to FDT
	8.5 Application: Modeling and Predicting Monsoon Intraseasonal Oscillation (MISO)
		8.5.1 MISO Indices from a Nonlinear Data Decomposition Technique
		8.5.2 Data-Driven Physics-Constrained Nonlinear Model
		8.5.3 Data Assimilation and Prediction
9 Parameter Estimation with Uncertainty Quantification
	9.1 Markov Chain Monte Carlo
		9.1.1 The Metropolis Algorithm
		9.1.2 A Simple Example
		9.1.3 Parameter Estimation via MCMC
		9.1.4 MCMC with Data Augmentation
	9.2 Expectation-Maximization
		9.2.1 The Mathematical Framework
		9.2.2 Details of the Quadratic Optimization in the Maximization Step
		9.2.3 Incorporating the Constraints into the EM Algorithm
		9.2.4 A Numerical Example
	9.3 Parameter Estimation via Data Assimilation
		9.3.1 Two Parameter Estimation Algorithms
		9.3.2 Estimating One Additive Parameter in a Linear Scalar Model
		9.3.3 Estimating One Multiplicative Parameter in a Linear Scalar Model
		9.3.4 Estimating Parameters in a Cubic Nonlinear Scalar Model
		9.3.5 A Numerical Example
	9.4 Learning Complex Dynamical Systems with Sparse Identification
		9.4.1 Constrained Optimization for Sparse Identification
		9.4.2 Using Information Theory for Model Identification with Sparsity
		9.4.3 Partial Observations and Stochastic Parameterizations
10 Combining Stochastic Models with Machine Learning
	10.1 Machine Learning
	10.2 Data Assimilation with Machine Learning Forecast Models
		10.2.1 Using Machine Learning as Surrogate Models for Data Assimilation
		10.2.2 Using Data Assimilation to Provide High-Quality Training Data  of Machine Learning Models
	10.3 Machine Learning for Stochastic Closure and Stochastic Parameterization
	10.4 Statistical Forecast Using Machine Learning
		10.4.1 Forecasting the Moment Equations with Neural Networks
		10.4.2 Incorporating Additional Forecast Uncertainty in the Machine Learning Path-Wise Forecast Results
11 Instruction Manual for the MATLAB Codes
	[DELETE]
References
-12pt
Index