Practical software testing: a process-oriented approach

Practical Software Testing......Page 1
CONTENTS......Page 6
Preface......Page 16
1.0 The Evolving Profession of Software Engineering......Page 24
1.1 The Role of Process in Software Quality......Page 27
1.2 Testing as a Process......Page 29
1.3 Overview of the Testing Maturity Model (TMM)......Page 31
1.3.1 TMM Levels......Page 33
Exercises......Page 39
References......Page 40
2.1 Basic Definitions......Page 42
2.2 Software Testing Principles......Page 49
2.3 The Tester\'s Role in a Software Development Organization......Page 57
Exercises......Page 59
References......Page 60
3.0 Origins of Defects......Page 62
3.1 Defect Classes, the Defect Repository, and Test Design......Page 66
3.1.1 Requirements and Specification Defects......Page 67
3.1.2 Design Defects......Page 69
3.1.3 Coding Defects......Page 71
3.2 Defect Examples: The Coin Problem......Page 74
3.3 Developer/Tester Support for Developing a Defect Repository......Page 80
Exercises......Page 81
References......Page 82
4.0 Introduction to Testing Design Strategies......Page 84
4.1 The Smart Tester......Page 85
4.2 Test Case Design Strategies......Page 86
4.4 Random Testing......Page 89
4.5 Equivalence Class Partitioning......Page 90
4.6 Boundary Value Analysis......Page 95
4.7 An Example of the Application of Equivalence Class Partitioning and Boundary Value Analysis......Page 96
4.8 Other Black Box Test Design Approaches......Page 99
4.8.1 Cause-and-Effect Graphing......Page 101
4.8.2 State Transition Testing......Page 105
4.8.3 Error Guessing......Page 108
4.9 Black Box Testing and Commercial Off-the-Shelf Components (COTS)......Page 109
4.10 Black Box Methods and TMM Level 2 Maturity Goals......Page 111
List of Key Terms......Page 114
Exercises......Page 115
References......Page 118
5.0 Using the White Box Approach to Test Design......Page 120
5.1 Test Adequacy Criteria......Page 121
5.2 Coverage and Control Flow Graphs......Page 124
5.3 Covering Code Logic......Page 126
5.4 Paths: Their Role in White Box–Based Test Design......Page 131
5.5.1 Data Flow and White Box Test Design......Page 134
5.5.2 Loop Testing......Page 138
5.5.3 Mutation Testing......Page 139
5.6 Evaluating Test Adequacy Criteria......Page 141
5.7 White Box Testing Methods and the TMM......Page 147
Exercises......Page 150
References......Page 153
6.0 The Need for Levels of Testing......Page 156
6.0.1 Levels of Testing and Software Development Paradigms......Page 158
6.1 Unit Test: Functions, Procedures, Classes, and Methods as Units......Page 160
6.2 Unit Test: The Need for Preparation......Page 161
6.3 Unit Test Planning......Page 162
6.4 Designing the Unit Tests......Page 164
6.5 The Class as a Testable Unit: Special Considerations......Page 165
6.6 The Test Harness......Page 171
6.7 Running the Unit Tests and Recording Results......Page 173
6.8 Integration Test: Goals......Page 175
6.9 Integration Strategies for Procedures and Functions......Page 176
6.10 Integration Strategies for Classes......Page 181
6.11 Designing Integration Tests......Page 182
6.12 Integration Test Planning......Page 185
6.13 System Test: The Different Types......Page 186
6.13.1 Functional Testing......Page 189
6.13.2 Performance Testing......Page 190
6.13.3 Stress Testing......Page 192
6.13.4 Configuration Testing......Page 194
6.13.5 Security Testing......Page 195
6.13.6 Recovery Testing......Page 198
6.15 Alpha, Beta, and Acceptance Tests......Page 199
6.16 Summary Statement on Testing Levels......Page 201
6.17 The Special Role of Use Cases......Page 202
6.18 Levels of Testing and the TMM......Page 204
Exercises......Page 207
References......Page 209
7.0 Introductory Concepts......Page 212
7.1 Testing and Debugging Goals and Policies......Page 214
7.2 Test Planning......Page 220
7.3 Test Plan Components......Page 223
7.4 Test Plan Attachments......Page 239
7.4.1 Test Design Specifications......Page 240
7.4.2 Test Case Specifications......Page 241
7.4.3 Test Procedure Specifications......Page 243
7.6 Reporting Test Results......Page 244
7.7 The Role of the Three Critical Groups in Test Planning and Policy Development......Page 249
7.8 Process and the Engineering Disciplines: The Role of the Individual as a Process Facilitator......Page 253
Exercises......Page 254
References......Page 255
8.0 Introducing the Test Specialist......Page 258
8.1 Skills Needed by a Test Specialist......Page 260
8.2 Building a Testing Group......Page 263
8.3 The Structure of the Testing Group......Page 265
8.4 The Technical Training Program......Page 270
8.5 Career Paths for Testers: An Industry Example......Page 273
8.6 Tester Certification......Page 275
8.7 Integrating Testing Activities in the Software Life Cycle......Page 276
8.8 The Test Organization, Technical Training Program, and Test Integration: Support from the Three Critical Views......Page 280
Exercises......Page 284
References......Page 285
9.0 Defining Terms......Page 286
9.1 Measurements and Milestones for Controlling and Monitoring......Page 289
9.1.1 Measurements for Monitoring Testing Status......Page 294
9.1.2 Measurements for Monitoring Tester Productivity......Page 298
9.1.3 Measurements for Monitoring Testing Costs......Page 299
9.1.4 Measurements for Monitoring Errors, Faults and Failures......Page 300
9.1.5 Monitoring Test Effectiveness......Page 302
9.2 Status Meetings, Reports, and Control Issues......Page 306
9.3 Criteria for Test Completion......Page 312
9.4 Software Configuration Management......Page 315
9.5 Controlling and Monitoring: Three Critical Views......Page 319
Exercises......Page 323
References......Page 325
10.0 Expanding the Testing Activity Umbrella......Page 326
10.1 Types of Reviews......Page 330
10.1.1 Inspections as a Type of Technical Review......Page 331
10.1.2 Walkthroughs as a Type of Technical Review......Page 333
10.2 Developing a Review Program......Page 334
10.3 The Need for Review Policies......Page 336
10.4 Components of Review Plans......Page 337
10.4.2 Preconditions and Items to be Reviewed......Page 338
10.4.3 Roles, Participants, Team Size, and Time Requirements......Page 340
10.4.5 Review Training......Page 343
10.4.6 Review Checklists......Page 347
10.5 Reporting Review Results......Page 356
10.7 Review Metrics......Page 360
10.9 The Self-Check or Personal Review......Page 363
10.10 Reviews and the TMM Critical Views......Page 366
Exercises......Page 368
References......Page 370
11.0 The Need for a Formal Test Measurement Program......Page 372
11.1 Some Measurement-Related Definitions......Page 376
11.2 Initiating a Measurement Program......Page 377
11.3 Software Quality Evaluation......Page 387
11.4 Measurement and TMM Levels......Page 395
11.4.1 Measurements for TMM Level 1......Page 396
11.4.2 Measurements for TMM Level 2......Page 398
11.4.3 Measurements for TMM Level 3......Page 400
11.4.4 Measurements for TMM Level 4......Page 404
11.4.5 Measurements for TMM Level 5......Page 406
11.5 A Test Measurement Program, Software Quality Valuations and the Three Critical Views......Page 409
Exercises......Page 412
References......Page 414
12.0 Review of Quality Concepts......Page 416
12.1 Quality Costs......Page 418
12.2 What Is Quality Control?......Page 420
12.3 The Role of Operational Profiles and Usage Models in Quality Control......Page 422
12.4 Support for Quality Control: Statistical Testing......Page 430
12.5 Software Reliability......Page 433
12.5.1 Measurements for Software Reliability......Page 436
12.6 Reliability, Quality Control, and Stop-Test Decisions......Page 437
12.6.1 Applying Reliability Models......Page 440
12.7 Confidence Levels and Quality Control......Page 445
12.8 Usability Testing and Quality Control......Page 447
12.9 An Approach to Usability Testing......Page 448
12.9.1 Exploratory Usability Testing......Page 449
12.9.3 Validation Usability Testing......Page 450
12.9.5 Usability Testing: Resource Requirements......Page 452
12.9.6 Usability Tests and Measurements......Page 453
12.10 Software Quality Control and the Three Critical Views......Page 456
Exercises......Page 459
References......Page 460
13.0 Processes and Defects......Page 462
13.1 History of Defect Analysis and Prevention......Page 464
13.2 Necessary Support for a Defect Prevention Program......Page 467
13.3 Techniques for Defect Analysis......Page 470
13.4 Defect Causal Analysis......Page 473
13.5 The Action Team: Making Process Changes......Page 477
13.6 Monitoring Actions and Process Changes......Page 480
13.7 Benefits of a Defect Prevention Program......Page 482
13.8 Defect Prevention and the Three Critical Views......Page 483
Exercises......Page 485
References......Page 486
14.0 Goals for the Testers\' Workbench......Page 488
14.1 Evaluating Testing Tools for the Workbench......Page 490
14.2 Tool Categories......Page 493
14.2.2 Tools for TMM Level 1......Page 495
14.2.3 TMM Level 2: Maturity Goals for Phase Definition......Page 497
14.2.4 Tools for Phase Definition......Page 498
14.2.5 TMM Level 3: Maturity Goals for Integration......Page 501
14.2.6 Tools for Integration......Page 503
14.2.7 TMM Level 4: Maturity Goals for Management and Measurement......Page 510
14.2.8 Tools for Management and Measurement......Page 512
14.2.9 TMM Level 5: Maturity Goals for Optimization/Defect Prevention/Quality Control......Page 515
14.2.10 Tools for Optimization/Defect Prevention/Quality Control......Page 517
14.3 The Testers\' Workbench and the Three Critical Views......Page 521
Exercises......Page 523
References......Page 524
15.0 TMM Maturity Goals: Support for a Quality Testing Process......Page 526
15.1 Process Engineering and Quality Control......Page 527
15.2 Fundamentals of Quantitative Process Control......Page 532
15.3 Activities for Quantitative Test Process Control......Page 535
15.4 Examples of the Application of Statistical Process Control......Page 539
15.5 Test Process Optimization: The Role of a Process Improvemen Group......Page 541
15.6 Technology Transfer......Page 546
15.7 Process Reuse......Page 549
15.7.1 Templates for Reusable Processes......Page 552
15.7.2 Procedures for Process Reuse......Page 554
15.8 Activities, Tasks and Responsibilities for Test Proces Control and Optimization......Page 556
Exercises......Page 558
References......Page 559
16.0 The Need for a Testing Maturity Model......Page 560
16.1 Approach to Model Development......Page 561
16.2 Process Improvement Model Representation......Page 566
16.3 The TMM Structure: The Testing Maturity Levels......Page 568
16.4 The TMM Assessment Model: Design Approach......Page 571
16.5.1 Assessment Team Selection and Training......Page 572
16.5.2 The Assessment Procedure......Page 574
16.5.3 The TMM Assessment Questionnaire......Page 579
16.6 The TMM Ranking Procedure......Page 581
16.7 Forms and Tools for Assessment Support......Page 585
16.8 Relationship of the TMM to Other Process Improvement Models......Page 586
16.9.1 TMM Application I: Evaluating the Usability of the TMM Questionnaire......Page 592
16.9.2 TMM Application II: Identifying Test Problem Areas and Risks......Page 595
16.9.3 TMM Application III: Software Test Consulting......Page 596
16.9.4 TMM Application IV: Role of Human Factors in Process Assessment......Page 599
16.9.5 Lessons Learned from the TMM Studies......Page 604
References......Page 606
APPENDIX I : TEST-RELATED REFERENCES......Page 610
APPENDIX I I : SAMPLE TEST PLAN......Page 634
Part 1: The TMM Questionnaire......Page 656
Section 1. Instructions for the Respondent......Page 657
Section 2. Respondent Identification and Background......Page 658
Section 3. Organizational Background......Page 660
Section 4. The TMM Questions......Page 662
Section 5. Testing Tool Questions......Page 682
Section 6. Testing Trends Questions......Page 685
Section 8. Glossary of TMM-Related Terms......Page 686
Part 2: TMM Activities, Tasks and Responsibilities......Page 693
Index......Page 724