Nonlinear Regression Modeling for Engineering Applications: Modeling, Model Validation, and Enabling Design of Experiments

Content: Series Preface xiii    Preface xv    Acknowledgments xxiii    Nomenclature xxv    Symbols xxxvii    Part I INTRODUCTION    1 Introductory Concepts 3    1.1 Illustrative Example     Traditional Linear Least-Squares Regression 3    1.2 How Models Are Used 7    1.3 Nonlinear Regression 7    1.4 Variable Types 8    1.5 Simulation 12    1.6 Issues 13    1.7 Takeaway 15    Exercises 15    2 Model Types 16    2.1 Model Terminology 16    2.2 A Classification of Mathematical Model Types 17    2.3 Steady-State and Dynamic Models 21    2.3.1 Steady-State Models 22    2.3.2 Dynamic Models (Time-Dependent, Transient) 24    2.4 Pseudo-First Principles     Appropriated First Principles 26    2.5 Pseudo-First Principles     Pseudo-Components 28    2.6 Empirical Models with Theoretical Grounding 28    2.6.1 Empirical Steady State 28    2.6.2 Empirical Time-Dependent 30    2.7 Empirical Models with No Theoretical Grounding 31    2.8 Partitioned Models 31    2.9 Empirical or Phenomenological? 32    2.10 Ensemble Models 32    2.11 Simulators 33    2.12 Stochastic and Probabilistic Models 33    2.13 Linearity 34    2.14 Discrete or Continuous 36    2.15 Constraints 36    2.16 Model Design (Architecture, Functionality, Structure) 37    2.17 Takeaway 37    Exercises 37    Part II PREPARATION FOR UNDERLYING SKILLS    3 Propagation of Uncertainty 43    3.1 Introduction 43    3.2 Sources of Error and Uncertainty 44    3.2.1 Estimation 45    3.2.2 Discrimination 45    3.2.3 Calibration Drift 45    3.2.4 Accuracy 45    3.2.5 Technique 46    3.2.6 Constants and Data 46    3.2.7 Noise 46    3.2.8 Model and Equations 46    3.2.9 Humans 47    3.3 Significant Digits 47    3.4 Rounding Off 48    3.5 Estimating Uncertainty on Values 49    3.5.1 Caution 50    3.6 Propagation of Uncertainty     Overview     Two Types, Two Ways Each 51    3.6.1 Maximum Uncertainty 51    3.6.2 Probable Uncertainty 56    3.6.3 Generality 58    3.7 Which to Report? Maximum or Probable Uncertainty 59    3.8 Bootstrapping 59    3.9 Bias and Precision 61    3.10 Takeaway 65    Exercises 66    4 Essential Probability and Statistics 67    4.1 Variation and Its Role in Topics 67    4.2 Histogram and Its PDF and CDF Views 67    4.3 Constructing a Data-Based View of PDF and CDF 70    4.4 Parameters that Characterize the Distribution 71    4.5 Some Representative Distributions 72    4.5.1 Gaussian Distribution 72    4.5.2 Log-Normal Distribution 72    4.5.3 Logistic Distribution 74    4.5.4 Exponential Distribution 74    4.5.5 Binomial Distribution 75    4.6 Confidence Interval 76    4.7 Central Limit Theorem 77    4.8 Hypothesis and Testing 78    4.9 Type I and Type II Errors, Alpha and Beta 80    4.10 Essential Statistics for This Text 82    4.10.1 t-Test for Bias 83    4.10.2 Wilcoxon Signed Rank Test for Bias 83    4.10.3 r-lag-1 Autocorrelation Test 84    4.10.4 Runs Test 87    4.10.5 Test for Steady State in a Noisy Signal 87    4.10.6 Chi-Square Contingency Test 89    4.10.7 Kolmogorov   Smirnov Distribution Test 89    4.10.8 Test for Proportion 90    4.10.9 F-Test for Equal Variance 90    4.11 Takeaway 91    Exercises 91    5 Simulation 93    5.1 Introduction 93    5.2 Three Sources of Deviation: Measurement, Inputs, Coefficients 93    5.3 Two Types of Perturbations: Noise (Independent) and Drifts (Persistence) 95    5.4 Two Types of Influence: Additive and Scaled with Level 98    5.5 Using the Inverse CDF to Generate n and u from UID(0, 1) 99    5.6 Takeaway 100    Exercises 100    6 Steady and Transient State Detection 101    6.1 Introduction 101    6.1.1 General Applications 101    6.1.2 Concepts and Issues in Detecting Steady State 104    6.1.3 Approaches and Issues to SSID and TSID 104    6.2 Method 106    6.2.1 Conceptual Model 106    6.2.2 Equations 107    6.2.3 Coefficient, Threshold, and Sample Frequency Values 108    6.2.4 Noiseless Data 111    6.3 Applications 112    6.3.1 Applications of the R-Statistic Approach for Process Monitoring 112    6.3.2 Applications of the R-Statistic Approach for Determining Regression Convergence 112    6.4 Takeaway 114    Exercises 114    Part III REGRESSION, VALIDATION, DESIGN    7 Regression Target     Objective Function 119    7.1 Introduction 119    7.2 Experimental and Measurement Uncertainty     Static and Continuous Valued 119    7.3 Likelihood 122    7.4 Maximum Likelihood 124    7.5 Estimating   x and   y Values 127    7.6 Vertical SSD     A Limiting Consideration of Variability Only in the Response Measurement 127    7.7 r-Square as a Measure of Fit 128    7.8 Normal, Total, or Perpendicular SSD 130    7.9 Akaho   s Method 132    7.10 Using a Model Inverse for Regression 134    7.11 Choosing the Dependent Variable 135    7.12 Model Prediction with Dynamic Models 136    7.13 Model Prediction with Classification Models 137    7.14 Model Prediction with Rank Models 138    7.15 Probabilistic Models 139    7.16 Stochastic Models 139    7.17 Takeaway 139    Exercises 140    8 Constraints 141    8.1 Introduction 141    8.2 Constraint Types 141    8.3 Expressing Hard Constraints in the Optimization Statement 142    8.4 Expressing Soft Constraints in the Optimization Statement 143    8.5 Equality Constraints 147    8.6 Takeaway 148    Exercises 148    9 The Distortion of Linearizing Transforms 149    9.1 Linearizing Coefficient Expression in Nonlinear Functions 149    9.2 The Associated Distortion 151    9.3 Sequential Coefficient Evaluation 154    9.4 Takeaway 155    Exercises 155    10 Optimization Algorithms 157    10.1 Introduction 157    10.2 Optimization Concepts 157    10.3 Gradient-Based Optimization 159    10.3.1 Numerical Derivative Evaluation 159    10.3.2 Steepest Descent     The Gradient 161    10.3.3 Cauchy   s Method 162    10.3.4 Incremental Steepest Descent (ISD) 163    10.3.5 Newton   Raphson (NR) 163    10.3.6 Levenberg   Marquardt (LM) 165    10.3.7 Modified LM 166    10.3.8 Generalized Reduced Gradient (GRG) 167    10.3.9 Work Assessment 167    10.3.10 Successive Quadratic (SQ) 167    10.3.11 Perspective 168    10.4 Direct Search Optimizers 168    10.4.1 Cyclic Heuristic Direct Search 169    10.4.2 Multiplayer Direct Search Algorithms 170    10.4.3 Leapfrogging 171    10.5 Takeaway 173    11 Multiple Optima 176    11.1 Introduction 176    11.2 Quantifying the Probability of Finding the Global Best 178    11.3 Approaches to Find the Global Optimum 179    11.4 Best-of-N Rule for Regression Starts 180    11.5 Interpreting the CDF 182    11.6 Takeaway 184    12 Regression Convergence Criteria 185    12.1 Introduction 185    12.2 Convergence versus Stopping 185    12.3 Traditional Criteria for Claiming Convergence 186    12.4 Combining DV Influence on OF 188    12.5 Use Relative Impact as Convergence Criterion 189    12.6 Steady-State Convergence Criterion 190    12.7 Neural Network Validation 197    12.8 Takeaway 198    Exercises 198    13 Model Design     Desired and Undesired Model Characteristics and Effects 199    13.1 Introduction 199    13.2 Redundant Coefficients 199    13.3 Coefficient Correlation 201    13.4 Asymptotic and Uncertainty Effects When Model is Inverted 203    13.5 Irrelevant Coefficients 205    13.6 Poles and Sign Flips w.r.t. the DV 206    13.7 Too Many Adjustable Coefficients or Too Many Regressors 206    13.8 Irrelevant Model Coefficients 215    13.8.1 Standard Error of the Estimate 216    13.8.2 Backward Elimination 216    13.8.3 Logical Tests 216    13.8.4 Propagation of Uncertainty 216    13.8.5 Bootstrapping 217    13.9 Scale-Up or Scale-Down Transition to New Phenomena 217    13.10 Takeaway 218    Exercises 218    14 Data Pre- and Post-processing 220    14.1 Introduction 220    14.2 Pre-processing Techniques 221    14.2.1 Steady- and Transient-State Selection 221    14.2.2 Internal Consistency 221    14.2.3 Truncation 222    14.2.4 Averaging and Voting 222    14.2.5 Data Reconciliation 223    14.2.6 Real-Time Noise Filtering for Noise Reduction (MA, FoF, STF) 224    14.2.7 Real-Time Noise filtering for Outlier Removal (Median Filter) 227    14.2.8 Real-Time Noise Filtering, Statistical Process Control 228    14.2.9 Imputation of Input Data 230    14.3 Post-processing 231    14.3.1 Outliers and Rejection Criterion 231    14.3.2 Bimodal Residual Distributions 233    14.3.3 Imputation of Response Data 235    14.4 Takeaway 235    Exercises 235    15 Incremental Model Adjustment 237    15.1 Introduction 237    15.2 Choosing the Adjustable Coefficient in Phenomenological Models 238    15.3 Simple Approach 238    15.4 An Alternate Approach 240    15.5 Other Approaches 241    15.6 Takeaway 241    Exercises 241    16 Model and Experimental Validation 242    16.1 Introduction 242    16.1.1 Concepts 242    16.1.2 Deterministic Models 244    16.1.3 Stochastic Models 246    16.1.4 Reality! 249    16.2 Logic-Based Validation Criteria 250    16.3 Data-Based Validation Criteria and Statistical Tests 251    16.3.1 Continuous-Valued, Deterministic, Steady State, or End-of-Batch 251    16.3.2 Continuous-Valued, Deterministic, Transient 263    16.3.3 Class/Discrete/Rank-Valued, Deterministic, Batch, or Steady State 264    16.3.4 Continuous-Valued, Stochastic, Batch, or Steady State 265    16.3.5 Test for Normally Distributed Residuals 266    16.3.6 Experimental Procedure Validation 266    16.4 Model Discrimination 267    16.4.1 Mechanistic Models 267    16.4.2 Purely Empirical Models 268    16.5 Procedure Summary 268    16.6 Alternate Validation Approaches 269    16.7 Takeaway 270    Exercises 270    17 Model Prediction Uncertainty 272    17.1 Introduction 272    17.2 Bootstrapping 273    17.3 Takeaway 276    18 Design of Experiments for Model Development and Validation 277    18.1 Concept     Plan and Data 277    18.2 Sufficiently Small Experimental Uncertainty     Methodology 277    18.3 Screening Designs     A Good Plan for an Alternate Purpose 281    18.4 Experimental Design     A Plan for Validation and Discrimination 282    18.4.1 Continually Redesign 282    18.4.2 Experimental Plan 283    18.5 EHS&LP 286    18.6 Visual Examples of Undesired Designs 287    18.7 Example for an Experimental Plan 289    18.8 Takeaway 291    Exercises 292    19 Utility versus Perfection 293    19.1 Competing and Conflicting Measures of Excellence 293    19.2 Attributes for Model Utility Evaluation 294    19.3 Takeaway 295    Exercises 296    20 Troubleshooting 297    20.1 Introduction 297    20.2 Bimodal and Multimodal Residuals 297    20.3 Trends in the Residuals 298    20.4 Parameter Correlation 298    20.5 Convergence Criterion     Too Tight, Too Loose 299    20.6 Overfitting (Memorization) 300    20.7 Solution Procedure Encounters Execution Errors 300    20.8 Not a Sharp CDF (OF) 300    20.9 Outliers 301    20.10 Average Residual Not Zero 302    20.11 Irrelevant Model Coefficients 302    20.12 Data Work-Up after the Trials 302    20.13 Too Many rs! 303    20.14 Propagation of Uncertainty Does Not Match Residuals 303    20.15 Multiple Optima 304    20.16 Very Slow Progress 304    20.17 All Residuals are Zero 304    20.18 Takeaway 305    Exercises 305    Part IV CASE STUDIES AND DATA    21 Case Studies 309    21.1 Valve Characterization 309    21.2 CO2 Orifice Calibration 311    21.3 Enrollment Trend 312    21.4 Algae Response to Sunlight Intensity 314    21.5 Batch Reaction Kinetics 316    Appendix A: VBA Primer: Brief on VBA Programming     Excel in Office 2013 319    Appendix B: Leapfrogging Optimizer Code for Steady-State Models 328    Appendix C: Bootstrapping with Static Model 341    References and Further Reading 350    Index 355