Systems Benchmarking: For Scientists And Engineers

Foreword by David Patterson......Page 5
Foreword by John R. Mashey......Page 7
Theme......Page 9
Goals, Prerequisites, and Target Audience......Page 11
Book Organization and Outline......Page 12
Acknowledgements......Page 16
Contents......Page 18
About the Authors......Page 25
Part I Foundations......Page 26
1.1 Definition of Benchmark......Page 27
1.2 System Quality Attributes......Page 29
1.3 Types of Benchmarks......Page 31
1.4.1 Fixed-Work Benchmarks......Page 33
14.2 Fixed-Time Benchmarks......Page 35
1.5 Benchmark Quality Criteria......Page 36
1.5.1 Relevance......Page 37
1.5.2 Reproducibility......Page 38
1.5.3 Fairness......Page 39
1.5.5 Usability......Page 41
1.6 Application Scenarios for Benchmarks......Page 42
1.7 Concluding Remarks......Page 43
References......Page 44
Chapter 2 Review of Basic Probability and Statistics......Page 46
2.1 Basic Concepts......Page 47
2.2 Distributions of Random Variables......Page 49
2.3 Independent and Dependent Random Variables......Page 51
2.4 Random Samples and Some Important Statistics......Page 53
2.5 Important Continuous Distributions and Central Limit Theorem......Page 55
2.6 The Bernoulli and Binomial Distributions......Page 59
2.7.1 Regression Analysis......Page 61
2.7.2 Kalman Filter......Page 62
2.7.4 Bayesian Inference......Page 64
2.7.5 Mathematical Optimization......Page 65
2.8 Concluding Remarks......Page 66
References......Page 67
3.1 Definition of Metric......Page 68
3.2 Scales of Measurement......Page 69
3.3.1 Speedup and Relative Change......Page 72
3.3.2 Basic Performance Metrics......Page 73
3.4 Quality Attributes of Good Metrics......Page 74
3.4.1 Motivating Example......Page 75
3.4.2 Quality Attributes......Page 78
3.5.1 Types of Averages......Page 80
3.5.2 Composite Metrics......Page 83
3.5.3 Aggregating Results from Multiple Benchmarks......Page 85
3.5.3.1 Aggregating Ratio Metrics......Page 86
3.5.3.2 Aggregating Normalized Values......Page 89
3.6 Concluding Remarks......Page 92
References......Page 93
Chapter 4 Statistical Measurements......Page 94
4.1 Measurement as a Random Experiment......Page 95
4.2.1 Experimental Errors......Page 97
4.2.2 A Model of Random Errors......Page 99
4.2.3 Estimating Means......Page 102
4.2.3.1 CI Based on the Normal Distribution......Page 103
4.2.3.2 CI Based on the......Page 105
4.2.3.3 Indirect Measurement......Page 109
4.2.3.4 Determining the Number of Measurements Needed......Page 110
4.2.4 Estimating Proportions......Page 111
4.2.4.1 Determining the Number of Measurements Needed......Page 112
4.3 Comparing Alternatives......Page 114
4.3.1 Non-corresponding Measurements......Page 115
4.3.2 Before-and-After Comparisons......Page 118
4.3.3 Comparing Proportions......Page 120
4.4 Concluding Remarks......Page 122
References......Page 123
Chapter 5 Experimental Design......Page 124
5.1 One-Factor Analysis of Variance......Page 125
5.2 Method of Contrasts......Page 130
5.3 Two-Factor Full Factorial Designs......Page 131
5.4 General m-Factor Full Factorial Designs......Page 137
5.5 Fractional Factorial Designs: Plackett–Burman......Page 140
5.6 Case Study......Page 144
5.6.1 Problem Statement......Page 146
5.6.2 Plackett–Burman Design......Page 147
5.6.4 Case Study Summary......Page 149
References......Page 151
6.1 Basic Measurement Strategies......Page 153
6.2 Interval Timers......Page 155
6.2.1 Timer Rollover......Page 156
6.2.2 Timer Accuracy......Page 157
6.2.3 Measuring Short Intervals......Page 159
6.3 Performance Profiling......Page 160
6.4 Event Tracing......Page 162
6.4.1 Call Path Tracing......Page 163
6.4.1.3 Loop Iteration Counts......Page 165
6.4.2 Performance Monitoring and Tracing Tools......Page 166
References......Page 167
Chapter 7 Operational Analysis and Basic Queueing Models......Page 170
7.1 Operational Analysis......Page 171
7.1.2 Service Demand Law......Page 173
7.1.3 Forced Flow Law......Page 174
7.1.4 Little's Law......Page 175
7.1.5 Interactive Response Time Law......Page 176
7.1.7 Performance Bounds......Page 178
7.2 Basic Queueing Theory......Page 181
7.2.1 Single Queues......Page 182
7.2.2 Queueing Networks......Page 184
7.2.4 Response Time Equations......Page 188
7.2.5 Solution Techniques for Queueing Networks......Page 189
7.2.6 Case Study......Page 190
7.2.6.1 Workload Characterization......Page 192
7.2.6.2 Building a Performance Model......Page 195
7.2.6.3 Model Analysis and Validation......Page 197
7.2.7 Conclusions from the Analysis......Page 201
7.3 Concluding Remarks......Page 202
References......Page 203
8.1 Workload Facets and Artifacts......Page 206
8.2 Executable Parts of a Workload......Page 208
8.3.1 Workload Traces......Page 210
8.3.2 Workload Descriptions......Page 212
8.3.2.1 ClosedWorkloads......Page 214
8.3.2.2 Open Workloads......Page 216
8.3.3 System-Metric-Based Workload Descriptions......Page 220
References......Page 221
9.1 Historical Perspective on Computer Systems Benchmarking......Page 223
9.2.1 SPEC's Origin......Page 226
9.2.3 Structure and Organization......Page 228
9.2.4 Open Systems Group (OSG)......Page 229
9.2.5 Graphics and Workstation Performance Group (GWPG)......Page 230
9.2.7 Research Group (RG)......Page 231
9.2.8 Benchmark Development Cycle......Page 232
9.3 Transaction Processing Performance Council (TPC)......Page 234
9.3.2 From Engineering to Marketing, From Benchmarking to Benchmarketing......Page 236
9.3.3 A Progression of Benchmarks......Page 237
9.3.4 Evolution of the TPC Model Over Time......Page 239
9.3.5 The Need and Opportunity for Kit-Based Benchmarks......Page 241
9.3.6 The Virtual World of Computing......Page 243
9.4 Concluding Remarks......Page 244
References......Page 245
Part II Applications......Page 248
10.1 SPEC CPU89......Page 249
10.2 SPEC CPU92......Page 251
10.3 SPEC CPU95......Page 254
10.4 SPEC CPU2000......Page 255
10.5 SPEC CPU2006......Page 256
10.6 SPEC CPU2017......Page 257
10.6.1 SPEC CPU2017 Suites......Page 258
10.6.2 SPEC CPU2017 Workloads......Page 259
10.6.3 SPEC CPU2017 Metrics......Page 261
10.6.4 SPEC CPU2017 Energy Metrics......Page 264
10.6.5 SPEC CPU2017 Full Disclosure Report......Page 265
References......Page 267
Chapter 11 Benchmarking the Energy Efficiency of Servers......Page 269
11.1.1 Device Setup......Page 270
11.1.3 Phases and Intervals......Page 272
11.1.4 Basic Energy-Efficiency Metric......Page 274
11.2.1 Metric Calculation......Page 275
11.3 SPEC Server Efficiency Rating Tool (SERT)......Page 276
11.3.1.1 CPU Workload......Page 277
11.3.1.4 IdleWorkload......Page 278
11.3.2.1 Interval Energy Efficiency......Page 279
11.3.2.4 Final Aggregate Energy Efficiency......Page 280
11.4 Concluding Remarks......Page 281
References......Page 282
Chapter 12 Virtualization Benchmarks......Page 284
12.1.1 SPEC VIRT_SC 2010......Page 285
12.1.1.1 Design......Page 286
12.1.1.2 The Need for a New Benchmark......Page 287
12.1.2 SPEC VIRT_SC 2013......Page 288
12.1.3.1 Design......Page 290
12.1.3.2 SPECvirt Datacenter 2020 Template VMs......Page 292
12.2.1 The VMmark 3 Benchmark......Page 293
12.2.1.1 Workloads......Page 294
12.2.1.2 Scoring Methodology......Page 299
References......Page 301
13.1 Historical Perspective on System File Server Benchmarks......Page 302
13.2 SPEC SFS 2014......Page 303
13.3.1 SPC-1......Page 307
13.3.3 Component and Energy Extension......Page 308
13.4 The IOzone Benchmark......Page 309
13.5 Flexible I/O Tester (fio)......Page 310
13.5.1 Running a Simple Job......Page 311
13.5.2 More Complex Workloads......Page 313
13.5.3 Unusual I/O Patterns......Page 315
13.5.4 ioengines......Page 316
References......Page 317
Chapter 14 TeaStore: A Microservice Reference Application......Page 318
14.1 Requirements on TeaStore......Page 319
14.2 Workload......Page 321
14.3 Architecture......Page 322
14.4.2 Image Provider......Page 325
14.4.4 Recommender......Page 326
14.5 Workload Descriptions for TeaStore......Page 327
14.6 Case Study: TeaStore in Action......Page 329
14.6.1 Energy Efficiency on Homogeneous Systems......Page 330
14.6.2 Energy Efficiency on Heterogeneous Systems......Page 332
References......Page 333
Chapter 15 Elasticity of Cloud Platforms......Page 336
15.1 Defining Elasticity......Page 337
15.2 Elasticity Metrics......Page 339
15.2.1 Provisioning Accuracy......Page 341
15.2.3 Instability......Page 342
15.3 Aggregating Elasticity Metrics......Page 344
15.3.1 Autoscaling Deviation......Page 345
15.3.3 Elastic Speedup Score......Page 346
15.4.1 Overview of Bungee......Page 347
15.5 Case Study......Page 350
References......Page 356
Chapter 16 Performance Isolation......Page 358
16.1 Definition of Performance Isolation......Page 359
16.2 Performance Isolation Metrics......Page 360
16.2.1 Metrics Goals and Requirements......Page 361
16.2.2 QoS-Impact-Based Isolation Metrics......Page 362
16.2.3 Workload-Ratio-Based Isolation Metrics......Page 364
16.2.3.1 Metrics Based on Edge Points......Page 366
16.2.3.2 Metrics Based on Integrals......Page 367
16.2.4.1 Settling Time......Page 368
16.2.4.2 Oscillation......Page 370
16.3.4.2 Unpinned Scenario......Page 375
16.3.5 Effectiveness of the Deployment Options......Page 376
16.3 Case Study......Page 371
16.3.3 Experimental Environment......Page 372
16.3.4 Performance Isolation Metrics in Action......Page 374
16.4.2 Edge-Point-Based Metrics......Page 377
16.4.4 Discussion......Page 378
References......Page 379
Chapter 17 Resource Demand Estimation......Page 382
17.1 Estimation Methods......Page 384
17.1.1 Approximation with Response Times......Page 385
17.1.3 Linear Regression......Page 386
17.1.5 Optimization......Page 387
17.1.6 Machine Learning......Page 388
17.1.8 Gibbs Sampling......Page 389
17.2 Input Parameters......Page 390
17.3 Output Metrics......Page 394
17.4 Robustness......Page 396
17.5 Estimation Accuracy and Execution Time......Page 398
17.6 Library for Resource Demand Estimation (LibReDE)......Page 400
References......Page 401
Chapter 18 Software and System Security......Page 406
18.1.1 Attacks and Common Security Mechanisms......Page 407
18.1.2 Application Scenarios......Page 409
18.2 Current State......Page 410
18.2.1 Workloads......Page 411
18.2.1.1 Workload Drivers......Page 412
18.2.1.3 Exploit Database......Page 413
18.2.1.4 Vulnerability and Attack Injection......Page 414
18.2.1.5 Trace Acquisition......Page 418
18.2.1.6 Trace Generation......Page 419
18.2.2 Metrics......Page 421
18.2.3.1 Attack Detection......Page 427
18.2.3.2 Resource Consumption......Page 429
18.2.3.4 Capacity......Page 430
18.3 Concluding Remarks......Page 431
References......Page 432
Index......Page 439