Nonlinear System Identification: From Classical Approaches to Neural Networks, Fuzzy Models, and Gaussian Processes

Preface to the Second Edition
Preface to the First Edition
Contents
Notation
1 Introduction
	1.1 Relevance of Nonlinear System Identification
		1.1.1 Linear or Nonlinear?
		1.1.2 Prediction
		1.1.3 Simulation
		1.1.4 Optimization
		1.1.5 Analysis
		1.1.6 Control
		1.1.7 Fault Detection
	1.2 Views on Nonlinear System Identification
	1.3 Tasks in Nonlinear System Identification
		1.3.1 Choice of the Model Inputs
		1.3.2 Choice of the Excitation Signals
		1.3.3 Choice of the Model Architecture
		1.3.4 Choice of the Dynamics Representation
		1.3.5 Choice of the Model Order
		1.3.6 Choice of the Model Structure and Complexity
		1.3.7 Choice of the Model Parameters
		1.3.8 Model Validation
		1.3.9 The Role of Fiddle Parameters
	1.4 White Box, Black Box, and Gray Box Models
	1.5 Outline of the Book and Some Reading Suggestions
	1.6 Terminology
Part I Optimization
	2 Introduction to Optimization
		2.1 Overview of Optimization Techniques
		2.2 Kangaroos
		2.3 Loss Functions for Supervised Methods
			2.3.1 Maximum Likelihood Method
			2.3.2 Maximum A Posteriori and Bayes Method
		2.4 Loss Functions for Unsupervised Methods
	3 Linear Optimization
		3.1 Least Squares (LS)
			3.1.1 Covariance Matrix of the Parameter Estimate
			3.1.2 Errorbars
			3.1.3 Orthogonal Regressors
			3.1.4 Regularization/Ridge Regression
				3.1.4.1 Efficient Computation
				3.1.4.2 Covariances for Ridge Regression
				3.1.4.3 Prior Parameters for Ridge Regression
			3.1.5 Ridge Regression: Alternative Formulation
			3.1.6 L1 Regularization
			3.1.7 Noise Assumptions
			3.1.8 Weighted Least Squares (WLS)
			3.1.9 Robust Regression
			3.1.10 Least Squares with Equality Constraints
			3.1.11 Smoothing Kernels
				3.1.11.1 Ridge Regression
			3.1.12 Effective Number of Parameters
			3.1.13 L2 Boosting
				3.1.13.1 Shrinkage
		3.2 Recursive Least Squares (RLS)
			3.2.1 Reducing the Computational Complexity
			3.2.2 Tracking Time-Variant Processes
			3.2.3 Relationship Between the RLS and the KalmanFilter
		3.3 Linear Optimization with Inequality Constraints
		3.4 Subset Selection
			3.4.1 Methods for Subset Selection
			3.4.2 Orthogonal Least Squares (OLS) for Forward Selection
			3.4.3 Ridge Regression or Subset Selection?
		3.5 Summary
		3.6 Problems
	4 Nonlinear Local Optimization
		4.1 Batch and Sample Adaptation
			4.1.1 Mini-Batch Adaptation
			4.1.2 Sample Adaptation
		4.2 Initial Parameters
		4.3 Direct Search Algorithms
			4.3.1 Simplex Search Method
			4.3.2 Hooke-Jeeves Method
		4.4 General Gradient-Based Algorithms
			4.4.1 Line Search
				4.4.1.1 Interval Reduction
				4.4.1.2 Interval Location
			4.4.2 Finite Difference Techniques
			4.4.3 Steepest Descent
			4.4.4 Newton's Method
			4.4.5 Quasi-Newton Methods
			4.4.6 Conjugate Gradient Methods
		4.5 Nonlinear Least Squares Problems
			4.5.1 Gauss-Newton Method
			4.5.2 Levenberg-Marquardt Method
		4.6 Constrained Nonlinear Optimization
		4.7 Summary
		4.8 Problems
	5 Nonlinear Global Optimization
		5.1 Simulated Annealing (SA)
		5.2 Evolutionary Algorithms (EA)
			5.2.1 Evolution Strategies (ES)
			5.2.2 Genetic Algorithms (GA)
			5.2.3 Genetic Programming (GP)
		5.3 Branch and Bound (B&B)
		5.4 Tabu Search (TS)
		5.5 Summary
		5.6 Problems
	6 Unsupervised Learning Techniques
		6.1 Principal Component Analysis (PCA)
		6.2 Clustering Techniques
			6.2.1 k-Means Algorithm
			6.2.2 Fuzzy C-Means (FCM) Algorithm
			6.2.3 Gustafson-Kessel Algorithm
			6.2.4 Kohonen's Self-Organizing Map (SOM)
			6.2.5 Neural Gas Network
			6.2.6 Adaptive Resonance Theory (ART) Network
			6.2.7 Incorporating Information About the Output
		6.3 Summary
		6.4 Problems
	7 Model Complexity Optimization
		7.1 Introduction
		7.2 Bias/Variance Tradeoff
			7.2.1 Bias Error
			7.2.2 Variance Error
			7.2.3 Tradeoff
				7.2.3.1 Dependency on the Amount of Data
				7.2.3.2 Optimism
		7.3 Evaluating the Test Error and Alternatives
			7.3.1 Training, Validation, and Test Data
			7.3.2 Cross-Validation (CV)
				7.3.2.1 S-Fold Cross-Validation
				7.3.2.2 Leave-One-Out Error
				7.3.2.3 Leave-One-Out Versus S-Fold CV
				7.3.2.4 Bootstrapping
				7.3.2.5 Why Ensemble Methods Work
			7.3.3 Information Criteria
				7.3.3.1 Effective Number of Parameters and Effective Amount of Data
			7.3.4 Multi-Objective Optimization
			7.3.5 Statistical Tests
			7.3.6 Correlation-Based Methods
		7.4 Explicit Structure Optimization
		7.5 Regularization: Implicit Structure Optimization
			7.5.1 Effective Parameters
			7.5.2 Regularization by Non-Smoothness Penalties
				7.5.2.1 Curvature Penalty
				7.5.2.2 Ridge Regression
				7.5.2.3 Weight Decay
			7.5.3 Regularization by Early Stopping
			7.5.4 Regularization by Constraints
			7.5.5 Regularization by Staggered Optimization
			7.5.6 Regularization by Local Optimization
		7.6 Structured Models for Complexity Reduction
			7.6.1 Curse of Dimensionality
			7.6.2 Hybrid Structures
				7.6.2.1 Parallel Model
				7.6.2.2 Series Model
				7.6.2.3 Parameter Scheduling Model
			7.6.3 Projection-Based Structures
			7.6.4 Additive Structures
			7.6.5 Hierarchical Structures
			7.6.6 Input Space Decomposition with Tree Structures
		7.7 Summary
		7.8 Problems
	8 Summary of Part I
Part II Static Models
	9 Introduction to Static Models
		9.1 Multivariable Systems
		9.2 Basis Function Formulation
			9.2.1 Global and Local Basis Functions
			9.2.2 Linear and Nonlinear Parameters
		9.3 Extended Basis Function Formulation
		9.4 Static Test Process
		9.5 Evaluation Criteria
	10 Linear, Polynomial, and Look-Up Table Models
		10.1 Linear Models
		10.2 Polynomial Models
			10.2.1 Regularized Polynomials
				10.2.1.1 Penalization of Offset
			10.2.2 Orthogonal Polynomials
			10.2.3 Summary Polynomials
		10.3 Look-Up Table Models
			10.3.1 One-Dimensional Look-Up Tables
			10.3.2 Two-Dimensional Look-Up Tables
			10.3.3 Optimization of the Heights
			10.3.4 Optimization of the Grid
			10.3.5 Optimization of the Complete Look-Up Table
			10.3.6 Incorporation of Constraints
				10.3.6.1 Constraints on the Grid
				10.3.6.2 Constraints on the Heights
			10.3.7 Properties of Look-Up Table Models
		10.4 Summary
		10.5 Problems
	11 Neural Networks
		11.1 Construction Mechanisms
			11.1.1 Ridge Construction
			11.1.2 Radial Construction
			11.1.3 Tensor Product Construction
		11.2 Multilayer Perceptron (MLP) Network
			11.2.1 MLP Neuron
			11.2.2 Network Structure
			11.2.3 Backpropagation
			11.2.4 MLP Training
				11.2.4.1 Initialization
				11.2.4.2 Regulated Activation Weight Neural Network (RAWN) or Extreme Learning Machine
				11.2.4.3 Nonlinear Optimization of the MLP
				11.2.4.4 Combined Training Methods for the MLP
			11.2.5 Simulation Examples
			11.2.6 MLP Properties
			11.2.7 Projection Pursuit Regression (PPR)
			11.2.8 Multiple Hidden Layers
			11.2.9 Deep Learning
		11.3 Radial Basis Function (RBF) Networks
			11.3.1 RBF Neuron
			11.3.2 Network Structure
			11.3.3 RBF Training
				11.3.3.1 Random Center Placement
				11.3.3.2 Clustering for Center Placement
				11.3.3.3 Complexity Controlled Clustering for Center Placement
				11.3.3.4 Grid-Based Center Placement
				11.3.3.5 Subset Selection for Center Placement
				11.3.3.6 Nonlinear Optimization for Center Placement
			11.3.4 Simulation Examples
			11.3.5 RBF Properties
			11.3.6 Regularization Theory
			11.3.7 Normalized Radial Basis Function (NRBF)Networks
				11.3.7.1 Training
				11.3.7.2 Side Effects of Normalization
				11.3.7.3 Properties
		11.4 Other Neural Networks
			11.4.1 General Regression Neural Network (GRNN)
			11.4.2 Cerebellar Model Articulation Controller(CMAC)
			11.4.3 Delaunay Networks
			11.4.4 Just-In-Time Models
		11.5 Summary
		11.6 Problems
	12 Fuzzy and Neuro-Fuzzy Models
		12.1 Fuzzy Logic
			12.1.1 Membership Functions
			12.1.2 Logic Operators
			12.1.3 Rule Fulfillment
			12.1.4 Accumulation
		12.2 Types of Fuzzy Systems
			12.2.1 Linguistic Fuzzy Systems
			12.2.2 Singleton Fuzzy Systems
			12.2.3 Takagi-Sugeno Fuzzy Systems
		12.3 Neuro-Fuzzy (NF) Networks
			12.3.1 Fuzzy Basis Functions
			12.3.2 Equivalence Between RBF Networks and Fuzzy Models
			12.3.3 What to Optimize?
				12.3.3.1 Optimization of the Consequent Parameters
				12.3.3.2 Optimization of the Premise Parameters
				12.3.3.3 Optimization of the Rule Structure
				12.3.3.4 Optimization of Operators
			12.3.4 Interpretation of Neuro-Fuzzy Networks
			12.3.5 Incorporating and Preserving Prior Knowledge
			12.3.6 Simulation Examples
		12.4 Neuro-Fuzzy Learning Schemes
			12.4.1 Nonlinear Local Optimization
			12.4.2 Nonlinear Global Optimization
			12.4.3 Orthogonal Least Squares Learning
			12.4.4 Fuzzy Rule Extraction by a Genetic Algorithm (FUREGA)
				12.4.4.1 Coding of the Rule Structure
				12.4.4.2 Overcoming the Curse of Dimensionality
				12.4.4.3 Nested Least Squares Optimization of the Singletons
				12.4.4.4 Constrained Optimization of the Input Membership Functions
				12.4.4.5 Application Example
			12.4.5 Adaptive Spline Modeling of Observation Data (ASMOD)
		12.5 Summary
		12.6 Problems
	13 Local Linear Neuro-Fuzzy Models: Fundamentals
		13.1 Basic Ideas
			13.1.1 Illustration of Local Linear Neuro-Fuzzy Models
			13.1.2 Interpretation of the Local Linear Model Offsets
				13.1.2.1 Advantages of Local Description
			13.1.3 Interpretation as Takagi-Sugeno Fuzzy System
			13.1.4 Interpretation as Extended NRBF Network
		13.2 Parameter Optimization of the Rule Consequents
			13.2.1 Global Estimation
			13.2.2 Local Estimation
			13.2.3 Global Versus Local Estimation
			13.2.4 Robust Regression
			13.2.5 Regularized Regression
			13.2.6 Data Weighting
		13.3 Structure Optimization of the Rule Premises
			13.3.1 Local Linear Model Tree (LOLIMOT) Algorithm
				13.3.1.1 The LOLIMOT Algorithm
				13.3.1.2 Computational Complexity
				13.3.1.3 Two Dimensions
				13.3.1.4 Convergence Behavior
				13.3.1.5 AICC
			13.3.2 Different Objectives for Structure and Parameter Optimization
			13.3.3 Smoothness Optimization
			13.3.4 Splitting Ratio Optimization
			13.3.5 Merging of Local Models
			13.3.6 Principal Component Analysis for Preprocessing
			13.3.7 Models with Multiple Outputs
		13.4 Summary
		13.5 Problems
	14 Local Linear Neuro-Fuzzy Models: Advanced Aspects
		14.1 Different Input Spaces for Rule Premises and Consequents
			14.1.1 Identification of Processes with Direction-Dependent Behavior
			14.1.2 Piecewise Affine (PWA) Models
		14.2 More Complex Local Models
			14.2.1 From Local Neuro-Fuzzy Models to Polynomials
			14.2.2 Local Quadratic Models for Input Optimization
				14.2.2.1 Local Sparse Quadratic Models
			14.2.3 Different Types of Local Models
		14.3 Structure Optimization of the Rule Consequents
		14.4 Interpolation and Extrapolation Behavior
			14.4.1 Interpolation Behavior
			14.4.2 Extrapolation Behavior
				14.4.2.1 Ensuring Interpretable Extrapolation Behavior
				14.4.2.2 Incorporation of Prior Knowledge into the Extrapolation Behavior
		14.5 Global and Local Linearization
		14.6 Online Learning
			14.6.1 Online Adaptation of the Rule Consequents
				14.6.1.1 Local Recursive Weighted Least Squares Algorithm
				14.6.1.2 How Many Local Models to Adapt
				14.6.1.3 Convergence Behavior
				14.6.1.4 Robustness Against Insufficient Excitation
				14.6.1.5 Parameter Variances and Blow-Up Effect
				14.6.1.6 Computational Effort
				14.6.1.7 Structure Mismatch
			14.6.2 Online Construction of the Rule PremiseStructure
		14.7 Oblique Partitioning
			14.7.1 Smoothness Determination
			14.7.2 Hinging Hyperplanes
			14.7.3 Smooth Hinging Hyperplanes
			14.7.4 Hinging Hyperplane Trees (HHT)
		14.8 Hierarchical Local Model Tree (HILOMOT) Algorithm
			14.8.1 Forming the Partition of Unity
			14.8.2 Split Parameter Optimization
				14.8.2.1 LOLIMOT Splits
				14.8.2.2 Local Model Center
				14.8.2.3 Convergence Behavior
			14.8.3 Building up the Hierarchy
			14.8.4 Smoothness Adjustment
			14.8.5 Separable Nonlinear Least Squares
				14.8.5.1 Idea
				14.8.5.2 Termination Criterion
				14.8.5.3 Constrained Optimization
				14.8.5.4 Robust Estimation
				14.8.5.5 Alternatives to Separable Nonlinear Least Squares
			14.8.6 Analytic Gradient Derivation
				14.8.6.1 Derivative of the Local Model Network
				14.8.6.2 Derivative of the Sigmoidal Splitting Function
				14.8.6.3 Derivative of the Local Model
				14.8.6.4 Summary
			14.8.7 Analyzing Input Relevance from Partitioning
				14.8.7.1 Relevance for One Split
				14.8.7.2 Relevance for the Whole Network
			14.8.8 HILOMOT Versus LOLIMOT
		14.9 Errorbars, Design of Excitation Signals,and Active Learning
			14.9.1 Errorbars
				14.9.1.1 Errorbars with Global Estimation
				14.9.1.2 Errorbars with Local Estimation
			14.9.2 Detecting Extrapolation
			14.9.3 Design of Excitation Signals
		14.10 Design of Experiments
			14.10.1 Unsupervised Methods
				14.10.1.1 Random
				14.10.1.2 Sobol Sequence
				14.10.1.3 Latin Hypercube (LH)
				14.10.1.4 Optimized Latin Hypercube
			14.10.2 Model Variance-Oriented Methods
				14.10.2.1 Optimal Design
				14.10.2.2 Polynomials
				14.10.2.3 Basis Function Network
				14.10.2.4 Multilayer Perceptron, Local Model Network, etc.
				14.10.2.5 Gaussian Process Regression
			14.10.3 Model Bias-Oriented Methods
				14.10.3.1 Model Committee
				14.10.3.2 Model Ensemble
				14.10.3.3 HILOMOT DoE
			14.10.4 Active Learning with HILOMOT DoE
				14.10.4.1 Active Learning in General
				14.10.4.2 Active Learning with HILOMOT DoE
				14.10.4.3 Query Optimization
				14.10.4.4 Sequential Strategy
				14.10.4.5 Comparison of HILOMOT DoE with Unsupervised Design
				14.10.4.6 Exploiting the Separation Between Premise and Consequent Input Spaces in Local Model Networks for DoE
				14.10.4.7 Semi-Batch Strategy
				14.10.4.8 Active Learning for Slow Modeling Approaches
				14.10.4.9 Applications of HILOMOT DoE
		14.11 Bagging Local Model Trees
			14.11.1 Unstable Models
			14.11.2 Bagging with HILOMOT
			14.11.3 Bootstrapping for Confidence Assessment
			14.11.4 Model Weighting
		14.12 Summary and Conclusions
		14.13 Problems
	15 Input Selection for Local Model Approaches
		15.1 Test Processes
			15.1.1 Test Process One (TP1)
			15.1.2 Test Process Two (TP2)
			15.1.3 Test Process Three (TP3)
			15.1.4 Test Process Four (TP4)
		15.2 Mixed Wrapper-Embedded Input Selection Approach: Authored by Julian Belz
			15.2.1 Investigation with Test Processes
				15.2.1.1 Test Process One
			15.2.2 Test Process Two
			15.2.3 Extensive Simulation Studies
				15.2.3.1 Evaluation Criteria
				15.2.3.2 Search Strategies
				15.2.3.3 A Priori Considerations
				15.2.3.4 Comparison Results
		15.3 Regularization-Based Input Selection Approach: Authored by Julian Belz
			15.3.1 Normalized L1 Split Regularization
			15.3.2 Investigation with Test Processes
				15.3.2.1 Test Process One
				15.3.2.2 Test Process Four
		15.4 Embedded Approach: Authored by Julian Belz
			15.4.1 Partition Analysis
			15.4.2 Investigation with Test Processes
				15.4.2.1 Test Process Three
				15.4.2.2 Test Process Two
		15.5 Visualization: Partial Dependence Plots
			15.5.1 Investigation with Test Processes
				15.5.1.1 Test Process One
				15.5.1.2 Test Process Two
		15.6 Miles per Gallon Data Set
			15.6.1 Mixed Wrapper-Embedded Input Selection
			15.6.2 Regularization-Based Input Selection
			15.6.3 Visualization: Partial Dependence Plot
			15.6.4 Critical Assessment of Partial Dependence Plots
	16 Gaussian Process Models (GPMs)
		16.1 Overview on Kernel Methods
			16.1.1 LS Kernel Methods
			16.1.2 Non-LS Kernel Methods
		16.2 Kernels
		16.3 Kernel Ridge Regression
			16.3.1 Transition to Kernels
		16.4 Regularizing Parameters and Functions
			16.4.1 Discrepancy in Penalty Terms
		16.5 Reproducing Kernel Hilbert Spaces (RKHS)
			16.5.1 Norms
			16.5.2 RKHS Objective and Solution
			16.5.3 Equivalent Kernels and Locality
			16.5.4 Two Points of View
				16.5.4.1 Similarity-Based View
				16.5.4.2 Superposition of Kernels View
		16.6 Gaussian Processes/Kriging
			16.6.1 Key Idea
			16.6.2 Some Basics
			16.6.3 Prior
			16.6.4 Posterior
			16.6.5 Incorporating Output Noise
			16.6.6 Model Variance
			16.6.7 Incorporating a Base Model
				16.6.7.1 Subsequent Optimization
				16.6.7.2 Simultaneous Optimization
			16.6.8 Relationship to RBF Networks
			16.6.9 High-Dimensional Kernels
		16.7 Hyperparameters
			16.7.1 Influence of the Hyperparameters
			16.7.2 Optimization of the Hyperparameters
				16.7.2.1 Number of Hyperparameters
				16.7.2.2 One Versus Multiple Length Scales
				16.7.2.3 Hyperparameter Optimization Methods
			16.7.3 Marginal Likelihood
				16.7.3.1 Likelihood for the Noise-Free Case
				16.7.3.2 Marginal Likelihood for the Noisy Case
				16.7.3.3 Marginal Likelihood Versus Leave-One-Out Cross Validation
			16.7.4 A Note on the Prior Variance
		16.8 Summary
		16.9 Problems
	17 Summary of Part II
Part III Dynamic Models
	18 Linear Dynamic System Identification
		18.1 Overview of Linear System Identification
		18.2 Excitation Signals
		18.3 General Model Structure
			18.3.1 Terminology and Classification
			18.3.2 Optimal Predictor
				18.3.2.1 Simulation
				18.3.2.2 Prediction
			18.3.3 Some Remarks on the Optimal Predictor
			18.3.4 Prediction Error Methods
		18.4 Time Series Models
			18.4.1 Autoregressive (AR)
			18.4.2 Moving Average (MA)
			18.4.3 Autoregressive Moving Average (ARMA)
		18.5 Models with Output Feedback
			18.5.1 Autoregressive with Exogenous Input (ARX)
				18.5.1.1 Least Squares (LS)
				18.5.1.2 Consistency Problem
				18.5.1.3 Instrumental Variables (IV) Method
				18.5.1.4 Correlation Functions Least Squares (COR-LS)
			18.5.2 Autoregressive Moving Average with Exogenous Input (ARMAX)
				18.5.2.1 Estimation of ARMAX Models
			18.5.3 Autoregressive Autoregressive with Exogenous Input (ARARX)
			18.5.4 Output Error (OE)
				18.5.4.1 Nonlinear Optimization of the OE Model Parameters
				18.5.4.2 Repeated Least Squares and Filtering for OE Model Estimation
			18.5.5 Box-Jenkins (BJ)
			18.5.6 State Space Models
			18.5.7 Simulation Example
		18.6 Models Without Output Feedback
			18.6.1 Finite Impulse Response (FIR)
				18.6.1.1 Comparison ARX Versus FIR
			18.6.2 Regularized FIR Models
				18.6.2.1 TC Kernel
				18.6.2.2 Filter Interpretation
			18.6.3 Bias and Variance of Regularized FIR Models
			18.6.4 Impulse Response Preservation (IRP) FIRApproach
				18.6.4.1 Impulse Response Preservation (IRP)
				18.6.4.2 Hyperparameter Optimization
				18.6.4.3 Order Selection
				18.6.4.4 Consequences of Undermodeling
				18.6.4.5 Summary
			18.6.5 Orthonormal Basis Functions (OBF)
				18.6.5.1 Laguerre Filters
				18.6.5.2 Poisson Filters
				18.6.5.3 Kautz Filters
				18.6.5.4 Generalized Filters
			18.6.6 Simulation Example
		18.7 Some Advanced Aspects
			18.7.1 Initial Conditions
			18.7.2 Consistency
			18.7.3 Frequency-Domain Interpretation
			18.7.4 Relationship Between Noise Model and Filtering
			18.7.5 Offsets
		18.8 Recursive Algorithms
			18.8.1 Recursive Least Squares (RLS) Method
			18.8.2 Recursive Instrumental Variables (RIV) Method
			18.8.3 Recursive Extended Least Squares (RELS)Method
			18.8.4 Recursive Prediction Error Methods (RPEM)
		18.9 Determination of Dynamic Orders
		18.10 Multivariable Systems
			18.10.1 P-Canonical Model
			18.10.2 Matrix Polynomial Model
			18.10.3 Subspace Methods
		18.11 Closed-Loop Identification
			18.11.1 Direct Methods
			18.11.2 Indirect Methods
				18.11.2.1 Two-Stage Method
				18.11.2.2 Coprime Factor Identification
			18.11.3 Identification for Control
		18.12 Summary
		18.13 Problems
	19 Nonlinear Dynamic System Identification
		19.1 From Linear to Nonlinear System Identification
		19.2 External Dynamics
			19.2.1 Illustration of the External Dynamics Approach
				19.2.1.1 Relationship Between the Input/Output Signals and the Approximator Input Space
				19.2.1.2 Principal Component Analysis and Higher-Order Differences
				19.2.1.3 One-Step Prediction Surfaces
				19.2.1.4 Effect of the Sampling Time
			19.2.2 Series-Parallel and Parallel Models
			19.2.3 Nonlinear Dynamic Input/Output Model Classes
				19.2.3.1 Models with Output Feedback
				19.2.3.2 Models Without Output Feedback
			19.2.4 Restrictions of Nonlinear Input/Output Models
		19.3 Internal Dynamics
		19.4 Parameter Scheduling Approach
		19.5 Training Recurrent Structures
			19.5.1 Backpropagation-Through-Time (BPTT)Algorithm
			19.5.2 Real-Time Recurrent Learning
		19.6 Multivariable Systems
			19.6.1 Issues with Multiple Inputs
				19.6.1.1 Asymmetry Going from ARX → OE to NARX → NOE
				19.6.1.2 Mixed Dynamic and Static Behavior
		19.7 Excitation Signals
			19.7.1 From PRBS to APRBS
				19.7.1.1 APRBS Construction
				19.7.1.2 APRBS: Smoothing the Steps
			19.7.2 Ramp
			19.7.3 Multisine
			19.7.4 Chirp
			19.7.5 APRBS
				19.7.5.1 Sinusoidal APRBS
			19.7.6 NARX and NOBF Input Spaces
			19.7.7 MISO Systems
				19.7.7.1 Excitation of One Input at a Time
				19.7.7.2 Excitation of All Inputs Simultaneously
				19.7.7.3 Hold Time
			19.7.8 Tradeoffs
		19.8 Optimal Excitation Signal Generator: Coauthored by Tim O. Heinz
			19.8.1 Approaches with Fisher Information
			19.8.2 Optimized Nonlinear Input Signal (OMNIPUS) for SISO Systems
			19.8.3 Optimized Nonlinear Input Signal (OMNIPUS) for MISO Systems
				19.8.3.1 Separate Optimization of Each Input
				19.8.3.2 Escaping the Curse of Dimensionality
				19.8.3.3 Results for Two Inputs
				19.8.3.4 Input Signal Correlation
				19.8.3.5 Input Value Distribution
				19.8.3.6 Extensions
		19.9 Determination of Dynamic Orders
		19.10 Summary
		19.11 Problems
	20 Classical Polynomial Approaches
		20.1 Properties of Dynamic Polynomial Models
		20.2 Kolmogorov-Gabor Polynomial Models
		20.3 Volterra-Series Models
		20.4 Parametric Volterra-Series Models
		20.5 NDE Models
		20.6 Hammerstein Models
		20.7 Wiener Models
		20.8 Problems
	21 Dynamic Neural and Fuzzy Models
		21.1 Curse of Dimensionality
			21.1.1 MLP Networks
			21.1.2 RBF Networks
			21.1.3 Singleton Fuzzy and NRBF Models
		21.2 Interpolation and Extrapolation Behavior
		21.3 Training
			21.3.1 MLP Networks
			21.3.2 RBF Networks
			21.3.3 Singleton Fuzzy and NRBF Models
		21.4 Integration of a Linear Model
		21.5 Simulation Examples
			21.5.1 MLP Networks
			21.5.2 RBF Networks
			21.5.3 Singleton Fuzzy and NRBF Models
		21.6 Summary
		21.7 Problems
	22 Dynamic Local Linear Neuro-Fuzzy Models
		22.1 One-Step Prediction Error Versus Simulation Error
		22.2 Determination of the Rule Premises
		22.3 Linearization
			22.3.1 Static and Dynamic Linearization
			22.3.2 Dynamics of the Linearized Model
			22.3.3 Different Rule Consequent Structures
		22.4 Model Stability
			22.4.1 Influence of Rule Premise Inputs on Stability
				22.4.1.1 Rule Premise Inputs Without Output Feedback
				22.4.1.2 Rule Premise Inputs with Output Feedback
			22.4.2 Lyapunov Stability and Linear MatrixInequalities (LMIs)
			22.4.3 Ensuring Stable Extrapolation
		22.5 Dynamic LOLIMOT Simulation Studies
			22.5.1 Nonlinear Dynamic Test Processes
			22.5.2 Hammerstein Process
			22.5.3 Wiener Process
			22.5.4 NDE Process
			22.5.5 Dynamic Nonlinearity Process
		22.6 Advanced Local Linear Methods and Models
			22.6.1 Local Linear Instrumental Variables (IV) Method
			22.6.2 Local Linear Output Error (OE) Models
			22.6.3 Local Linear ARMAX Models
		22.7 Local Regularized Finite Impulse Response Models: Coauthored by Tobias Münker
			22.7.1 Structure
			22.7.2 Local Estimation
			22.7.3 Hyperparamter Tuning
			22.7.4 Evaluation of Performance
		22.8 Local Linear Orthonormal Basis Functions Models
		22.9 Structure Optimization of the Rule Consequents
		22.10 Summary and Conclusions
		22.11 Problems
	23 Neural Networks with Internal Dynamics
		23.1 Fully Recurrent Networks
		23.2 Partially Recurrent Networks
		23.3 State Recurrent Networks
		23.4 Locally Recurrent Globally Feedforward Networks
		23.5 Long Short-Term Memory (LSTM) Networks
		23.6 Internal Versus External Dynamics
		23.7 Problems
Part IV Applications
	24 Applications of Static Models
		24.1 Driving Cycle
			24.1.1 Process Description
			24.1.2 Smoothing of a Driving Cycle
			24.1.3 Improvements and Extensions
			24.1.4 Differentiation
			24.1.5 The Role of Look-Up Tables in Automotive Electronics
			24.1.6 Modeling of Exhaust Gases
			24.1.7 Optimization of Exhaust Gases
			24.1.8 Outlook: Dynamic Models
		24.2 Summary
	25 Applications of Dynamic Models
		25.1 Cooling Blast
			25.1.1 Process Description
			25.1.2 Experimental Results
				25.1.2.1 Excitation Signal Design
				25.1.2.2 Modeling and Identification
		25.2 Diesel Engine Turbocharger
			25.2.1 Process Description
			25.2.2 Experimental Results
				25.2.2.1 Excitation and Validation Signals
				25.2.2.2 Modeling and Identification
				25.2.2.3 Model Properties
				25.2.2.4 Choice of Sampling Time
		25.3 Thermal Plant
			25.3.1 Process Description
			25.3.2 Transport Process
				25.3.2.1 Modeling and Identification
				25.3.2.2 Model Properties
			25.3.3 Tubular Heat Exchanger
				25.3.3.1 Modeling and Identification
				25.3.3.2 Model Properties
			25.3.4 Cross-Flow Heat Exchanger
				25.3.4.1 Data
				25.3.4.2 Modeling and Identification
				25.3.4.3 Model Properties
		25.4 Summary
	26 Design of Experiments
		26.1 Practical DoE Aspects: Authored by Julian Belz
			26.1.1 Function Generator
			26.1.2 Order of Experimentation
			26.1.3 Biggest Gap Sequence
			26.1.4 Median Distance Sequence
			26.1.5 Intelligent k-Means Sequence
				26.1.5.1 Intelligent k-Means Initialization
			26.1.6 Other Determination Strategies
			26.1.7 Comparison on Synthetic Functions
			26.1.8 Summary
			26.1.9 Corner Measurement
			26.1.10 Comparison of Space-Filling Designs
		26.2 Active Learning for Structural Health Monitoring
			26.2.1 Simulation Results
			26.2.2 Experimental Results
		26.3 Active Learning for Engine Measurement
			26.3.1 Problem Setting
			26.3.2 Operating Point-Specific Engine Models
				26.3.2.1 Results
				26.3.2.2 Reducing Measurement Time with HILOMOT DoE
				26.3.2.3 Multiple Outputs
			26.3.3 Global Engine Model
		26.4 Nonlinear Dynamic Excitation Signal Design for Common Rail Injection
			26.4.1 Example: High-Pressure Fuel Supply System
			26.4.2 Identifying the Rail Pressure System
				26.4.2.1 Local Model Networks
				26.4.2.2 Gaussian Process Models (GPMs)
			26.4.3 Results
				26.4.3.1 Operating Point Depending Constraints
				26.4.3.2 Data Acquisition
				26.4.3.3 Accuracy of the Simulation Results
				26.4.3.4 Qualitative Analysis
				26.4.3.5 Quantitative Analysis
				26.4.3.6 Data Coverage of the Input Space
	27 Input Selection Applications
		27.1 Air Mass Flow Prediction
			27.1.1 Mixed Wrapper-Embedded Input Selection
			27.1.2 Partition Analysis
		27.2 Fan Metamodeling: Authored by Julian Belz
			27.2.1 Centrifugal Impeller Geometry
			27.2.2 Axial Impeller Geometry
			27.2.3 Why Metamodels?
			27.2.4 Design of Experiments: Centrifugal FanMetamodel
			27.2.5 Design of Experiments: Axial Fan Metamodel
			27.2.6 Order of Experimentation
			27.2.7 Goal-Oriented Active Learning
			27.2.8 Mixed Wrapper-Embedded Input Selection
			27.2.9 Centrifugal Fan Metamodel
			27.2.10 Axial Fan Metamodel
			27.2.11 Summary
		27.3 Heating, Ventilating, and Air Conditioning System
			27.3.1 Problem Configuration
			27.3.2 Available Data Sets
			27.3.3 Mixed Wrapper-Embedded Input Selection
			27.3.4 Results
	28 Applications of Advanced Methods
		28.1 Nonlinear Model Predictive Control
		28.2 Online Adaptation
			28.2.1 Variable Forgetting Factor
			28.2.2 Control and Adaptation Models
			28.2.3 Parameter Transfer
			28.2.4 Systems with Multiple Inputs
			28.2.5 Experimental Results
		28.3 Fault Detection
			28.3.1 Methodology
			28.3.2 Experimental Results
		28.4 Fault Diagnosis
			28.4.1 Methodology
			28.4.2 Experimental Results
		28.5 Reconfiguration
	29 LMN Toolbox
		29.1 Termination Criteria
			29.1.1 Corrected AIC
			29.1.2 Corrected BIC
			29.1.3 Validation
			29.1.4 Maximum Number of Local Models
			29.1.5 Effective Number of Parameters
			29.1.6 Maximum Training Time
		29.2 Polynominal Degree of Local Models
		29.3 Dynamic Models
			29.3.1 Nonlinear Orthonormal Basis Function Models
		29.4 Different Input Spaces x and z
		29.5 Smoothness
		29.6 Data Weighting
		29.7 Visualization and Simplified Tool
A Vectors and Matrices
	A.1 Vector and Matrix Derivatives
	A.2 Gradient, Hessian, and Jacobian
B Statistics
	B.1 Deterministic and Random Variables
	B.2 Probability Density Function (pdf)
	B.3 Stochastic Processes and Ergodicity
	B.4 Expectation
	B.5 Variance
	B.6 Correlation and Covariance
	B.7 Properties of Estimators
References
Index