Verification and Validation in Scientific Computing

Cover......Page 1
Half-title......Page 3
Title......Page 5
Copyright......Page 6
Dedication......Page 7
Contents......Page 9
Preface......Page 13
Acknowledgments......Page 15
1.1.1 Historical role of modeling and simulation......Page 17
1.1.2.1 Changing role of scientific computing in design, performance and safety of engineering systems......Page 19
1.1.2.2 Interaction of scientific computing and experimental investigations......Page 21
1.2.1 Computer speed and maturity of scientific computing......Page 24
1.2.2 Perspectives on credibility of scientific computing......Page 26
1.2.3.1 Quality of the analysts conducting the scientific computing......Page 28
1.2.3.3 Verification and validation activities......Page 29
1.2.3.4 Uncertainty quantification and sensitivity analyses......Page 30
1.3.1 Structure of the book......Page 31
1.3.2 Use of the book in undergraduate and graduate courses......Page 32
1.4 References......Page 33
Part I Fundamental concepts......Page 35
2.1 Development of concepts and terminology......Page 37
2.1.1 Early efforts of the operations research community......Page 38
2.1.2 IEEE and related communities......Page 39
2.1.4 AIAA and ASME communities......Page 41
2.1.4.1 AIAA Guide......Page 42
2.1.4.2 ASME Guide......Page 45
2.1.5 Hydrology community......Page 47
2.2.1 Code verification......Page 48
2.2.2 Solution verification......Page 50
2.2.3 Model validation......Page 51
2.2.4 Predictive capability......Page 55
2.2.5 Calibration......Page 60
2.2.6 Certification and accreditation......Page 63
2.3.1 Aleatory uncertainty......Page 67
2.3.2 Epistemic uncertainty......Page 69
2.3.2.1 Recognized uncertainty......Page 71
2.3.2.2 Blind Uncertainty......Page 72
2.4 Error in a quantity......Page 73
2.5 Integration of verification, validation, and prediction......Page 75
2.5.1 Specification of the application of interest......Page 76
2.5.2 Planning and prioritization of activities......Page 77
2.5.3 Code verification and software quality assurance activities......Page 79
2.5.4 Design and execution of validation experiments......Page 80
2.5.5 Computation of the system response quantities and solution verification......Page 82
2.5.6 Computation of validation metric results......Page 83
2.5.7 Prediction and uncertainty estimation for the application of interest......Page 85
2.5.8 Assessment of model adequacy......Page 88
2.5.9 Documentation of M&S activities......Page 89
2.6 References......Page 91
3 Modeling and computational simulation......Page 99
3.1.1 Systems and surroundings......Page 100
Example 1: Orbiting spacecraft......Page 101
Example 3: Electronic circuit......Page 102
3.1.2 Environments and scenarios......Page 103
3.2.1 Goals of scientific computing......Page 105
3.2.2 Models and simulations......Page 108
3.2.3 Importance of nondeterministic simulations......Page 112
3.2.4 Analysis of nondeterministic systems......Page 113
3.2.5.1 Aleatory uncertainty......Page 117
3.2.5.2 Epistemic uncertainty......Page 124
3.3 Risk and failure......Page 131
3.4 Phases of computational simulation......Page 132
3.4.1 Conceptual modeling phase......Page 134
3.4.2 Mathematical modeling phase......Page 135
3.4.3 Discretization and algorithm selection phase......Page 137
3.4.4 Computer programming phase......Page 139
3.4.5 Numerical solution phase......Page 140
3.4.6 Solution representation phase......Page 141
3.5 Example problem: missile flight dynamics......Page 143
3.5.1 Conceptual modeling phase......Page 144
3.5.2 Mathematical modeling phase......Page 148
3.5.4 Computer programming phase......Page 151
3.5.6 Solution representation phase......Page 152
3.6 References......Page 153
Part II Code verification......Page 161
4 Software engineering......Page 162
4.1.1 Software process models......Page 163
4.1.3 Programming languages......Page 164
4.1.4 Agile programming......Page 166
4.2 Version control......Page 167
4.3.1 Definitions......Page 169
4.3.2.2 Compiling the code......Page 170
Unit testing......Page 171
Component testing......Page 172
4.3.3.3 Software validation testing......Page 173
4.3.5 Code coverage......Page 174
4.4 Software quality and reliability......Page 175
4.4.1.2 Complexity analysis......Page 176
4.5 Case study in reliability: the T experiments......Page 177
4.6.1 Software requirements......Page 178
4.6.1.2 Requirements engineering process......Page 179
4.6.2.1 Project management......Page 180
4.6.2.3 Configuration management......Page 181
4.6.2.5 Process improvement......Page 182
4.7 References......Page 183
5 Code verification......Page 186
5.1 Code verification criteria......Page 187
5.1.1.2 Conservation tests......Page 188
5.1.2 Code-to-code comparisons......Page 189
5.1.5 Order-of-accuracy tests......Page 190
5.2.1 Truncation error......Page 191
5.2.1.1 Example: truncation error analysis......Page 192
5.2.1.2 Generalized truncation error expression (GTEE)......Page 193
5.2.3 Consistency......Page 194
5.2.5 Convergence......Page 195
5.3.1 Formal order of accuracy......Page 196
5.3.2 Observed order of accuracy......Page 198
5.4 Systematic mesh refinement......Page 201
5.4.2 Consistent mesh refinement......Page 202
5.4.3 Mesh transformations......Page 205
5.4.4 Mesh topology issues......Page 206
5.5 Order verification procedures......Page 208
2 Choose numerical algorithm......Page 209
3 Establish formal order of accuracy......Page 210
6 Compute observed order of accuracy......Page 211
5.5.2 Temporal discretization......Page 212
5.5.3.1 Separate order analysis......Page 213
5.5.3.2 Combined order analysis......Page 214
5.5.5 Limitations of order verification......Page 217
5.5.6.1 Residual method......Page 218
5.5.6.3 Downscaling method......Page 219
5.6 Responsibility for code verification......Page 220
5.7 References......Page 221
6 Exact solutions......Page 224
6.1 Introduction to differential equations......Page 225
6.2 Traditional exact solutions......Page 226
6.2.1.1 Separation of variables......Page 227
6.2.1.2 Transformations......Page 228
6.2.1.3 Method of characteristics......Page 229
6.2.3 Example with order verification: steady Burgers’ equation......Page 230
6.2.4 Example with order verification: linear elasticity......Page 233
6.3 Method of manufactured solutions (MMS)......Page 235
6.3.1 Procedure......Page 236
6.3.1.1 Manufactured solution guidelines for code verification......Page 238
6.3.1.2 Boundary and initial conditions......Page 239
6.3.2 Benefits of MMS for code verification......Page 241
6.3.3 Limitations of MMS for code verification......Page 242
6.3.4.1 2-D steady heat conduction......Page 244
6.3.4.2 2D Steady Euler equations......Page 246
6.4 Physically realistic manufactured solutions......Page 250
6.4.2 Method of nearby problems (MNP)......Page 251
6.4.2.1 Procedure......Page 252
6.4.2.2 Example exact solution: 2-D steady Navier–Stokes equations......Page 254
6.5 Approximate solution methods......Page 255
6.5.2 Reduction to ordinary differential equations......Page 256
6.5.4 Example series solution: 2-D steady heat conduction......Page 257
6.5.5 Example benchmark convergence test: 2-D hypersonic flow......Page 259
6.6 References......Page 260
Part III Solution verification......Page 265
7.1 Elements of solution verification......Page 266
7.2.1 Floating point representation......Page 268
7.2.2 Specifying precision in a code......Page 270
7.2.2.1 C/C++ programming languages......Page 271
7.2.2.2 Fortran 95/2003 programming languages......Page 272
7.2.2.3 MATLAB® Programming Language......Page 273
7.3 Statistical sampling error......Page 274
7.3.1 Estimation of statistical sampling error......Page 275
7.4 Iterative error......Page 276
7.4.1 Iterative methods......Page 277
7.4.1.1 Equations with a single unknown......Page 278
7.4.1.2 Systems of equations......Page 279
Direct solution methods......Page 281
Stationary iterative methods......Page 282
Krylov subspace methods......Page 284
Examples of iterative methods in scientific computing......Page 285
7.4.2 Iterative convergence......Page 289
Monotone convergence......Page 290
General convergence......Page 291
7.4.2.2 Iterative convergence criteria......Page 292
Iterative residuals......Page 293
7.4.3 Iterative error estimation......Page 294
Monotone iterative convergence......Page 295
7.4.4 Relation between iterative residuals and iterative error......Page 297
7.4.5 Practical approach for estimating iterative error......Page 298
7.5 Numerical error versus numerical uncertainty......Page 299
7.6 References......Page 300
8 Discretization error......Page 302
8.1.1 Discretization of the mathematical model......Page 304
8.1.1.1 The finite difference method......Page 305
8.1.1.2 The finite volume method......Page 306
8.1.1.3 The finite element method......Page 308
8.1.2.1 Structured meshes......Page 310
8.1.2.3 Cartesian meshes......Page 311
8.1.2.4 Mesh-free methods......Page 312
8.2 Approaches for estimating discretization error......Page 313
8.2.1.1 Mesh refinement methods......Page 315
8.2.1.3 Finite element recovery methods......Page 316
8.2.2.1 Error transport equations......Page 317
Continuous discretization error transport equation......Page 318
Approximating the truncation error......Page 319
System response quantities......Page 320
Implicit residual methods......Page 321
8.2.2.3 Adjoint methods for system response quantities......Page 322
Adjoint methods in the finite volume method......Page 324
8.3 Richardson extrapolation......Page 325
8.3.1 Standard Richardson extrapolation......Page 326
8.3.2 Generalized Richardson extrapolation......Page 327
8.3.3.2 Uniform mesh spacing......Page 328
8.3.4 Extensions......Page 329
8.3.4.3 Least squares extrapolation......Page 330
8.3.5.1 Example: Richardson extrapolation-based error estimation......Page 331
8.3.6 Advantages and disadvantages......Page 332
8.4.1 Asymptotic range......Page 333
8.4.2.1 Constant grid refinement factor......Page 334
8.4.2.3 Application to system response quantities......Page 336
8.4.2.4 Application to local quantities......Page 337
8.5 Discretization error and uncertainty......Page 338
8.6 Roache’s grid convergence index (GCI)......Page 339
8.6.1 Definition......Page 340
8.6.2 Implementation......Page 341
8.6.3.1 Least squares method......Page 342
8.6.3.2 Global averaging method......Page 343
8.6.3.3 Factor of safety method......Page 344
8.7.1 Measuring systematic mesh refinement......Page 345
8.7.2 Grid refinement factor......Page 346
8.7.3 Fractional uniform refinement......Page 347
8.7.4 Refinement vs. coarsening......Page 348
8.7.5 Unidirectional refinement......Page 349
8.8.1 Singularities and discontinuities......Page 350
8.8.4 Coarse grid error estimators......Page 353
8.9 References......Page 354
9.1 Factors affecting the discretization error......Page 359
9.1.2 1-D truncation error analysis on uniform meshes......Page 360
9.1.3 1-D truncation error analysis on nonuniform meshes......Page 361
9.1.4 Isotropic versus anisotropic mesh adaptation......Page 363
9.2.2 Discretization error......Page 365
9.2.3 Recovery methods......Page 366
9.2.4 Truncation errorresiduals......Page 367
9.2.4.2 Finite element residual-based methods......Page 368
9.2.5 Adjoint-based adaptation......Page 370
9.3 Adaptation approaches......Page 372
9.3.2.1 Local mesh refinementcoarsening (h-adaptation)......Page 374
9.3.2.2 Mesh movement (r-adaptation)......Page 375
9.4 Comparison of methods for driving mesh adaptation......Page 376
9.4.2 Exact solution......Page 377
9.4.3 Discretization approach......Page 378
9.4.4 Results......Page 379
9.5 References......Page 382
Part IV Model validation and prediction......Page 385
10 Model validation fundamentals......Page 395
10.1.1 Validation experiments vs. traditional experiments......Page 396
10.1.2 Goals and strategy of validation......Page 398
10.1.2.1 Scientific validation......Page 399
10.1.2.2 Project-oriented validation......Page 402
10.1.3 Sources of error in experiments and simulations......Page 405
10.1.4 Validation using data from traditional experiments......Page 409
10.2 Validation experiment hierarchy......Page 412
10.2.1 Characteristics of the complete system tier......Page 414
10.2.3 Characteristics of the benchmark tier......Page 415
10.2.4 Characteristics of the unit problem tier......Page 416
10.2.5 Construction of a validation hierarchy......Page 418
10.3 Example problem: hypersonic cruise missile......Page 420
10.3.2 Subsystem tier......Page 421
10.3.3 Benchmark tier......Page 422
10.3.5 Validation pyramid......Page 423
10.3.6 Final comments......Page 424
10.4.1 Conceptual difficulties......Page 425
10.4.2 Technical and practical difficulties......Page 428
10.5 References......Page 429
11.1 Guidelines for validation experiments......Page 433
11.1.1 Joint effort between analysts and experimentalists......Page 434
11.1.2 Measurement of all needed input data......Page 436
11.1.3 Synergism between computation and experiment......Page 439
11.1.4 Independence and dependence between computation and experiment......Page 440
11.1.5 Hierarchy of experimental measurements......Page 442
11.1.6 Estimation of experimental uncertainty......Page 445
11.2.1 Basic goals and description of JCEAP......Page 446
11.2.2.1 Wind tunnel conditions......Page 447
11.2.2.2 Model geometry......Page 449
11.2.2.3 Model fabrication and instrumentation......Page 451
11.2.3 Characterize boundary conditions and system data......Page 453
11.2.4 Synergism between computation and experiment......Page 456
11.2.6 Hierarchy of experimental measurements......Page 458
11.3.1 Random and systematic uncertainties......Page 461
11.3.2.1 DOE principles......Page 466
11.3.2.2 DOE analysis and results......Page 468
11.3.3.1 DOE principles......Page 472
11.3.3.2 DOE analysis and results......Page 474
11.4 Example of further computational–experimental synergism in JCEAP......Page 479
11.4.1 Assessment of computational submodels......Page 480
11.4.1.3 Thermodynamic submodel......Page 481
11.4.1.5 Outflow boundary condition assumption......Page 482
11.4.1.7 Re-evaluation of the experimental data......Page 483
11.4.2.1 Use of the flowfield calibration data......Page 484
11.4.2.2 Simulation using the nonuniform flowfield......Page 486
11.4.3 Lessons learned for validation experiments......Page 488
11.5 References......Page 489
12 Model accuracy assessment......Page 493
12.1 Elements of model accuracy assessment......Page 494
12.1.1 Methods of comparing simulations and experiments......Page 495
12.1.2 Uncertainty and error in model accuracy assessment......Page 498
12.1.3 Relationship between model accuracy assessment, calibration, and prediction......Page 500
12.2 Approaches to parameter estimation and validation metrics......Page 503
12.2.2 Hypothesis testing......Page 504
12.2.3 Bayesian updating......Page 507
12.2.4 Comparison of mean values......Page 508
12.3.1 Influence of numerical solution error......Page 510
12.3.3 Inclusion of experimental data post-processing......Page 511
12.3.4 Inclusion of experimental uncertainty estimation......Page 512
12.3.6 Exclusion of any type of adequacy implication......Page 513
12.3.7 Properties of a mathematical metric......Page 514
12.4.1 Perspectives of the present approach......Page 515
12.4.2 Development of the fundamental equations......Page 517
12.4.3 Construction of the validation metric for one condition......Page 519
12.4.4 Example problem: thermal decomposition of foam......Page 521
12.5.1 Construction of the validation metric over the range of the data......Page 524
12.5.2 Global metrics......Page 525
12.5.3 Example problem: turbulent buoyant plume......Page 526
12.6.1 Construction of the validation metric over the range of the data......Page 532
12.6.3 Example problem: thermal decomposition of foam......Page 534
12.7.2 Computation of simultaneous confidence intervals for the metric......Page 538
12.7.3 Global metrics......Page 540
12.7.4.1 Problem description......Page 541
12.7.4.3 Mathematical model......Page 542
12.7.4.4 Validation metric results......Page 544
12.7.5 Observations on the present approach......Page 547
12.8 Validation metric for comparing p-boxes......Page 548
12.8.1 Traditional methods for comparing distributions......Page 549
12.8.2.1 Discussion of p-boxes......Page 550
12.8.2.2 Validation metric for p-boxes......Page 552
12.8.3.1 u-pooling......Page 559
12.8.3.2 Statistical significance of a metric......Page 563
12.8.4 Inconsistency between experimental and simulation CDFs......Page 564
12.8.5.2 Epistemic and aleatory uncertainty in the metric......Page 568
12.9 References......Page 572
13 Predictive capability......Page 579
13.1 Step 1: identify all relevant sources of uncertainty......Page 581
13.1.1 Model inputs......Page 582
13.1.2 Model uncertainty......Page 584
13.1.3 Example problem: heat transfer through a plate......Page 585
13.2 Step 2: characterize each source of uncertainty......Page 589
13.2.1 Model input uncertainty......Page 593
13.2.2 Model uncertainty......Page 596
13.2.3.1 Model input uncertainty......Page 599
13.2.3.2 Model uncertainty......Page 604
13.3 Step 3: estimate numerical solution error......Page 608
13.3.1.2 Practical difficulties......Page 610
13.3.2.1 Temporal discretization error......Page 612
13.3.2.3 Richardson extrapolation error estimators for mesh convergence......Page 613
13.3.2.4 Practical difficulties......Page 614
13.3.3 Estimate of total numerical solution error......Page 615
13.3.4.1 Iterative and discretization error estimation......Page 617
13.3.4.2 Iterative and discretization error results......Page 621
13.4 Step 4: estimate output uncertainty......Page 623
13.4.1 Monte Carlo sampling of input uncertainties......Page 624
13.4.1.1 Monte Carlo sampling for aleatory uncertainties......Page 625
13.4.1.2 Monte Carlo sampling for combined aleatory and epistemic uncertainties......Page 630
13.4.2 Combination of input, model, and numerical uncertainty......Page 634
13.4.2.1 Combination of input and model uncertainty......Page 635
13.4.2.2 Estimation of model uncertainty using alternative plausible models......Page 639
13.4.3 Example problem: heat transfer through a solid......Page 641
13.4.3.1 Input uncertainties......Page 642
13.4.3.2 Combination of input, model, and numerical uncertainties......Page 644
13.5.1 Types of model parameter......Page 646
13.5.2 Sources of new information......Page 649
13.5.3 Approaches to parameter updating......Page 650
13.5.4 Parameter updating, validation, and predictive uncertainty......Page 652
13.5.4.1 Parameter updating......Page 653
13.5.4.2 Validation after parameter updating......Page 655
13.6 Step 6: conduct sensitivity analysis......Page 657
13.6.1 Local sensitivity analysis......Page 658
13.6.2 Global sensitivity analysis......Page 659
13.7 Example problem: thermal heating of a safety component......Page 662
13.7.2.1 Model input uncertainty......Page 664
Possible temperature dependence of material properties......Page 667
Characterization of model uncertainty......Page 669
13.7.3.1 General discussion of combining input and model uncertainty......Page 679
13.7.3.2 Combining input and model uncertainty for the thermal heating problem......Page 682
13.7.3.3 Predicted probabilities for the regulatory condition......Page 686
13.8 Bayesian approach as opposed to PBA......Page 688
13.9 References......Page 689
Part V Planning, management, and implementation issues......Page 695
14.1 Methodology for planning and prioritization......Page 697
14.1.1 Planning for a modeling and simulation project......Page 698
14.1.2 Value systems for prioritization......Page 700
14.2 Phenomena identification and ranking table (PIRT)......Page 702
14.2.1.2 Definition of the objectives of the PIRT process......Page 703
14.2.1.3 Specification of environments and scenarios......Page 704
14.2.1.4 Identification of plausible physical phenomena......Page 705
14.2.1.5 Construction of the PIRT......Page 706
14.3 Gap analysis process......Page 708
14.3.1 Construct the gap analysis table......Page 709
14.3.2 Documenting the PIRT and gap analysis processes......Page 712
14.3.3 Updating the PIRT and gap analysis......Page 713
14.4 Planning and prioritization with commercial codes......Page 714
14.5 Example problem: aircraft fire spread during crash landing......Page 715
14.6 References......Page 718
15.1 Survey of maturity assessment procedures......Page 720
15.2.1 Structure of the PCMM......Page 726
15.2.1.1 Representation and geometric fidelity......Page 727
15.2.1.2 Physics and material model fidelity......Page 728
15.2.1.3 Maturity assessment......Page 729
15.2.2 Purpose and uses of the PCMM......Page 731
15.2.3.1 Representation and geometric fidelity......Page 735
15.2.3.2 Physics and material model fidelity......Page 738
15.2.3.3 Code verification......Page 739
15.2.3.4 Solution verification......Page 740
15.2.3.5 Model validation......Page 742
15.2.3.6 Uncertainty quantification and sensitivity analysis......Page 743
15.3.1 Requirements for modeling and simulation maturity......Page 745
15.3.2 Aggregation of PCMM scores......Page 747
15.3.3 Use of the PCMM in risk-informed decision making......Page 748
15.4 References......Page 749
16.1 Needed technical developments......Page 752
16.2.1.1 Who should conduct SQA and code verification?......Page 753
16.2.1.2 Who should require SQA and code verification?......Page 755
16.2.2.1 Who should conduct solution verification?......Page 756
16.2.2.2 Who should require solution verification?......Page 757
16.2.3.1 Who should conduct validation?......Page 758
16.2.3.2 Who should require validation?......Page 759
16.2.4.1 Who should conduct nondeterministic predictions?......Page 760
16.2.4.2 Who should require nondeterministic predictions?......Page 761
16.3.1 Implementation issues......Page 762
16.3.3 Incorporation into business goals......Page 765
16.3.3.1 Intrinsic information quality......Page 766
16.3.3.2 Contextual information quality......Page 767
16.3.3.3 Representational information quality......Page 768
16.3.4 Organizational structures......Page 769
16.4 Development of databases......Page 771
16.4.1 Existing databases......Page 772
16.4.2 Recent activities......Page 773
16.4.3 Implementation issues of Databases......Page 774
16.5 Development of standards......Page 777
16.6 References......Page 779
Use static analyzers......Page 781
Use indentation for readability......Page 782
Duplicate code......Page 783
GOTO statements......Page 784
References......Page 785
Index......Page 786