GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs (GAO-09-3SP).

Preface......Page 3
Abbreviations......Page 17
Introduction......Page 19
The Guide’s Readers......Page 21
Acknowledgments......Page 22
The Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them......Page 23
Table 1: GAO’s 1972 Version of the Basic Characteristics of Credible Cost Estimates......Page 24
Case Study 2: Basic Estimate Characteristics, from Customs Service Modernization, GAO/AIMD-99-41......Page 26
Why Government Programs Need Cost Estimates and the Challenges in Developing Them......Page 33
Figure 2: Challenges Cost Estimators Typically Face......Page 35
Case Study 12: Applying EVM, from Cooperative Threat Reduction, GAO-06-692......Page 41
Criteria for Cost Estimating, EVM, and Data Reliability......Page 43
Determining Data Reliability......Page 47
Main Cost Estimate Categories......Page 49
The Overall Significance of Cost Estimates......Page 54
Cost Estimates in Acquisition......Page 55
Figure 4: Cone of Uncertainty......Page 56
Case Study 14: Realistic Estimates, from Defense Acquisitions, GAO-05-183......Page 57
Case Study 16: Importance of Realistic LCCEs, from Space Acquisitions, GAO-07-96......Page 60
Scope......Page 65
Schedule......Page 66
Figure 9: Disciplines and Concepts in Cost Analysis......Page 69
Team Composition and Organization......Page 70
Cost Estimating Team Best Practices......Page 71
Table 5: Certification Standards in Business, Cost Estimating, and Financial Management in the Defense Acquisition Education, Training, and Career Development Program......Page 72
Process......Page 75
Table 6: Typical Technical Baseline Elements......Page 76
Table 7: General System Characteristics......Page 79
Best Practice: Product-Oriented WBS......Page 83
Table 8: Common Elements in Work Breakdown Structures......Page 86
Standardized WBS......Page 89
WBS and Risk Management......Page 93
WBS Benefits......Page 94
Assumptions......Page 97
Case Study 25: The Importance of Assumptions, from Space Acquisitions, GAO-07-96......Page 98
Assumptions, Sensitivity, and Risk Analysis......Page 100
Data Collection......Page 107
Case Study 31: Fitting the Estimating Approach to the Data, from Space Acquisitions, GAO-07-96......Page 109
Table 10: Basic Primary and Secondary Data Sources......Page 110
Data Applicability......Page 113
Validating and Analyzing the Data......Page 114
EVM Data Reliability......Page 116
Data Normalization......Page 118
Recurring and Nonrecurring Costs......Page 119
Case Study 33: Inflation, from Defense Acquisitions, GAO-05-183......Page 120
Selecting the Proper Indexes......Page 121
Data Documentation......Page 122
Cost Estimating Methods......Page 125
Production Rate Effects on Learning......Page 139
Case Study 36: Production Rate, from Defense Acquisitions, GAO-05-183......Page 140
Case Study 37: Underestimating Software, from Space Acquisitions, GAO-07-96......Page 143
Table 15: The Twelve Steps of High-Quality Cost Estimating Summarized......Page 144
Table 16: Sizing Metrics and Commonly Associated Issues......Page 145
Estimating Software Development Effort......Page 149
Table 17: Common Software Risks That Affect Cost and Schedule......Page 151
Software Maintenance......Page 152
Parametric Software Estimation......Page 153
Commercial Off-the-Shelf Software......Page 154
Enterprise Resource Planning Software......Page 155
Unique Components of IT Estimation......Page 157
Sensitivity Analysis......Page 165
Sensitivity Factors......Page 166
Case Study 38: Sensitivity Analysis, from Defense Acquisitions, GAO-05-183......Page 167
Figure 14: A Sensitivity Analysis That Creates a Range around a Point Estimate......Page 168
The Difference Between Risk and Uncertainty......Page 171
Point Estimates Alone Are Insufficient for Good Decisions......Page 172
Figure 15: Changes in Cost Estimate Uncertainty across the Acquisition Life Cycle......Page 173
Developing a Credible S Curve of Potential Program Costs......Page 177
Risk Management......Page 193
The Cost Estimating Community’s Best Practices for Validating Estimates......Page 197
Documenting the Estimate......Page 209
Table 28: What Cost Estimate Documentation Includes......Page 211
Other Considerations......Page 213
Presenting the Estimate to Management......Page 215
Figure 20: Integrating Cost Estimation, Systems Development Oversight, and Risk Management......Page 217
The Nature and History of EVM......Page 220
Implementing EVM......Page 222
Federal and Industry Guidelines for Implementing EVM......Page 229
The Thirteen Steps in the EVM Process......Page 233
Integrated Baseline Reviews......Page 249
Award Fees......Page 251
Progress and Performance-Based Payments Under Fixed-Price Contracts......Page 252
Figure 31: The EVM System Acceptance Process......Page 254
Validating the Performance Measurement Baseline with an IBR......Page 261
Contract Performance Reports......Page 267
Case Study 46: Cost Performance Reports, from Defense Acquisitions, GAO-05-183......Page 273
Project Future Performance......Page 283
Figure 39: Rolling Wave Planning......Page 287
Incorporating Authorized Changes into the Performance Measurement Baseline......Page 291
Using EVM System Surveillance to Keep the Performance Measurement Baseline Current......Page 294
Overtarget Baselines and Schedules......Page 302
Update the Program Cost Estimate with Actual Costs......Page 311
Keep Management Updated......Page 312
Auditing Agencies and Their Web Sites......Page 315
Table 48: Case Studies Drawn from GAO Reports Illustrating This Guide......Page 316
Experts Who Helped Develop This Guide......Page 323
Table 49: Phases of the Budget Process......Page 326
Federal Cost Estimating and EVM Legislation, Regulations, Policies, and Guidance......Page 328
Data Collection Instrument......Page 339
Data Collection Instrument: Data Request Rationale......Page 341
SEI Checklist......Page 345
Table 51: Aircraft System Work Breakdown Structure......Page 346
Schedule Risk Analysis......Page 371
Learning Curve Analysis......Page 385
Technology Readiness Levels......Page 391
EVM-Related Award Fee Criteria......Page 392
Table 72: IBR Leadership Roles and Responsibilities......Page 395
Exhibit A – Program X: Subject Contractor......Page 400
Exhibit B – IBR Discussion Assessment Forms......Page 401
Exhibit C – IBR Action Item Forms......Page 403
Exhibit D – Sample IBR Discussion Questions......Page 405
Common Risks to Consider in Software Cost Estimating......Page 411
Contacts and Acknowledgments......Page 422
References......Page 423
Image Sources......Page 438
Figure 3: Life-Cycle Cost Estimate for a Space System......Page 51
Figure 5: An Affordability Assessment......Page 58
Case Study 15: Importance of Realistic LCCEs, from Combating Nuclear Smuggling, GAO-07-133R......Page 59
Case Study 17: Evolutionary Acquisition and Cost Estimates, from Best Practices, GAO-03-645T......Page 61
Case Study 18: Incremental Development, from Customs Service Modernization, GAO/AIMD-99-41......Page 62
Figure 10: A Product-Oriented Work Breakdown Structure......Page 84
Figure 12: A Contract Work Breakdown Structure......Page 88
Figure 13: A Learning Curve......Page 138
Figure 16: A Cumulative Probability Distribution, or S Curve......Page 175
Figure 17: A Risk Cube Two-Dimensional Matrix......Page 182
Figure 18: The Distribution of Sums from Rolling Two Dice......Page 185
Table 24: Eight Common Probability Distributions......Page 186
Figure 21: Integrating EVM and Risk Management......Page 219
Table 30: Ten Common Concerns about EVM......Page 225
Figure 23: WBS Integration of Cost, Schedule, and Technical Information......Page 234
Figure 24: Identifying Responsibility for Managing Work at the Control Account......Page 235
Figure 25: An Activity Network......Page 237
Figure 26: Activity Durations as a Gantt Chart......Page 239
Figure 27: Earned Value, Using the Percent Complete Method, Compared to Planned Costs......Page 245
Figure 28: The Genesis of the Performance Measurement Baseline......Page 246
Figure 29: The Time-Phased Cumulative Performance Measurement Baseline......Page 247
Figure 30: A Performance-Based Payments Structured Contract......Page 253
Figure 32: IBR Control Account Manager Discussion Template......Page 266
Figure 33: Monthly Program Assessment Using Earned Value......Page 268
Figure 34: Overall Program View of EVM Data......Page 269
Figure 35: A Contract Performance Report’s Five Formats......Page 270
Figure 36: Understanding Program Cost Growth by Plotting Budget at Completion Trends......Page 279
Figure 37: Understanding Program Performance by Plotting Cost and Schedule Variances......Page 280
Figure 38: Understanding Expected Cost Overruns by Plotting Estimate at Completion......Page 281
Case Study 48: Maintaining Realistic Baselines, from Uncertainties Remain, GAO-04-643R......Page 303
Figure 41: Steps Typically Associated with Implementing an Overtarget Budget......Page 307
Figure 42: Establishing a New Baseline with a Single Point Adjustment......Page 309
Figure 43: MasterFormat™ Work Breakdown Structure......Page 369
Figure 44: Network Diagram of a Simple Schedule......Page 372
Figure 45: Example Project Schedule......Page 373
Figure 46: Estimated Durations for Remaining WBS Areas in the Schedule......Page 374
Figure 48: Identified Risks on a Spacecraft Schedule: An Example......Page 375
Figure 50: Spacecraft Schedule Results from a Monte Carlo Simulation......Page 376
Figure 51: A Schedule Showing Critical Path through Unit 2......Page 377
Figure 53: Sensitivity Index for Spacecraft Schedule......Page 378
Figure 55: An Example of Probabilistic Branching Contained in the Schedule......Page 379
Figure 56: Probability Distribution Results for Probabilistic Branching in Test Unit......Page 380
Figure 58: Risk Results Assuming No Correlation between Activity Durations......Page 381
Figure 60: Schedule Analysis Results with and without Correlation......Page 382
Figure 62: Program X IBR Team’s Assessment Results by Program Area......Page 397
Figure 63: Program X IBR Team’s Detailed Assessment Results for an Individual Program Area......Page 398
Table 2: The Twelve Steps of a High-Quality Cost Estimating Process......Page 27
Table 3: Cost Estimating and EVM Criteria for Federal Agencies: Legislation, Regulations, Policies, and Guidance......Page 44
Table 4: Life-Cycle Cost Estimates, Types of Business Case Analyses, and Other Types of Cost Estimates......Page 52
Table 9: Cost Element Structure for a Standard DOD Automated Information System......Page 90
Table 11: Three Cost Estimating Methods Compared......Page 126
Table 12: An Example of the Analogy Cost Estimating Method......Page 127
Table 13: An Example of the Engineering Build-Up Cost Estimating Method......Page 129
Table 14: An Example of the Parametric Cost Estimating Method......Page 131
Table 18: Best Practices Associated with Risks in Implementing ERP......Page 156
Table 19: Common IT Infrastructure Risks......Page 160
Table 20: Common Labor Categories Described......Page 161
Table 21: Potential Sources of Program Cost Estimate Uncertainty......Page 178
Table 22: A Hardware Risk Scoring Matrix......Page 183
Table 23: A Software Risk Scoring Matrix......Page 184
Table 25: The Twelve Steps of High-Quality Cost Estimating, Mapped to the Characteristics of a High-Quality Cost Estimate......Page 198
Table 26: Questions for Checking the Accuracy of Estimating Techniques......Page 201
Table 27: Eight Types of Independent Cost Estimate Reviews......Page 206
Table 29: Key Benefits of Implementing EVM......Page 224
Table 31: ANSI Guidelines for EVM Systems......Page 230
Table 32: EVM Implementation Guides......Page 232
Table 33: Typical Methods for Measuring Earned Value Performance......Page 243
Table 34: Integrated Baseline Review Risk Categories......Page 264
Table 35: Contract Performance Report Data Elements: Format 1......Page 271
Table 36: EVM Performance Indexes......Page 277
Table 37: Best Predictive EAC Efficiency Factors by Program Completion Status......Page 284
16.Best Practices Checklist: Managing Program Costs: Execution......Page 288
Table 39: ANSI Guidelines Related to Incorporating Changes in an EVM System......Page 292
Table 40: Elements of an Effective Surveillance Organization......Page 295
Table 42: Risk Factors That Warrant EVM Surveillance......Page 296
Table 43: A Program Surveillance Selection Matrix......Page 297
Table 44: A Color-Category Rating System for Summarizing Program Findings......Page 301
Table 45: Overtarget Budget Funding Implications by Contract Type......Page 304
Table 46: Common Indicators of Poor Program Performance......Page 305
Table 47: Options for Treating Variances in Performing a Single Point Adjustment......Page 308
Table 50: The Budget Process: Major Steps and Time Periods......Page 327
Table 52: Electronic/Automated Software System Work Breakdown Structure......Page 347
Table 53: Ground Software Work Breakdown Structure......Page 348
Table 54: Missile System Work Breakdown Structure......Page 349
Table 55: Ordnance System Work Breakdown Structure......Page 351
Table 56: Sea System Work Breakdown Structure......Page 352
Table 57: Space System Work Breakdown Structure......Page 353
Table 58: Surface Vehicle System Work Breakdown Structure......Page 354
Table 59: Unmanned Air Vehicle System Work Breakdown Structure......Page 355
Table 60: Department of Energy Project Work Breakdown Structure......Page 357
Table 61: General Services Administration Construction Work Breakdown Structure......Page 358
Table 62: Automated Information System: Enterprise Resource Planning Program Level Work Breakdown Structure......Page 359
Table 64: Pharmaceutical Work Breakdown Structure......Page 360
Table 66: Telecon Work Breakdown Structure......Page 362
Table 67: Software Implementation Project Work Breakdown Structure......Page 363
Table 68: Major Renovation Project Work Breakdown Structure......Page 364
Table 69: Sample IT Infrastructure and Service Work Breakdown Structure......Page 365
Table 70: CSI MasterFormatTM 2004 Structure Example: Construction Phase......Page 370
Table 71: The Anderlohr Method for the Learning Lost Factor......Page 388
Case Study 1: Basic Estimate Characteristics, from NASA, GAO-04-642......Page 25
Case Study 3: Following Cost Estimating Steps, from NASA, GAO-04-642......Page 30
Case Study 5: Recognizing Uncertainty, from Customs Service Modernization, GAO/AIMD-99-41......Page 36
Case Study 6: Using Realistic Assumptions, from Space Acquisitions, GAO-07-96......Page 37
Case Study 8: Program Stability Issues, from Defense Acquisitions, GAO-05-183......Page 38
Case Study 10: Risk Analysis, from Defense Acquisitions, GAO-05-183......Page 39
Case Study 11: Risk Analysis, from NASA, GAO-04-642......Page 40
Case Study 13: Rebaselining, from NASA, GAO-04-642......Page 42
2.Best Practices Checklist: Purpose, Scope, and Schedule......Page 67
Case Study 20: Defining Requirement, from United States Coast Guard, GAO-06-623......Page 78
Case Study 21: Managing Requirements, from DOD Systems Modernization, GAO-06-215......Page 81
Case Study 22: Product-Oriented Work Breakdown Structure, from Air Traffic Control, GAO-08-756......Page 85
Case Study 24: Developing Work Breakdown Structure, from Homeland Security, GAO-06-296......Page 95
Case Study 26: Testing Ground Rules for Risk, from Space Acquisitions, GAO-07-96......Page 102
Case Study 27: The Industrial Base, from Defense Acquisition, GAO-05-183......Page 103
Case Study 29: Technology Maturity, from Space Acquisitions, GAO-07-96......Page 104
Case Study 30: Informing Management of Changed Assumptions, from Customs Service Modernization, GAO/AIMD-99-41......Page 105
Case Study 32: Data Anomalies, from Cooperative Threat Reduction, GAO-06-692......Page 117
Case Study 34: Cost Estimating Methods, from Space Acquisitions, GAO-07-96......Page 128
Case Study 35: Expert Opinion, from Customs Service Modernization, GAO/AIMD-99-41......Page 136
Case Study 40: Point Estimates, from Defense Acquisitions, GAO-05-183......Page 174
Case Study 41: Validating the Estimate, from Chemical Demilitarization, GAO-07-240R......Page 199
Case Study 42: Independent Cost Estimates, from Space Acquisitions, GAO-07-96......Page 205
Case Study 43: Documenting the Estimate, from Telecommunications, GAO-07-268......Page 210
Case Study 44: Validating the EVM System, from Cooperative Threat Reduction, GAO-06-692......Page 256
Case Study 45: Validating the EVM System, from DOD Systems Modernization, GAO-06-215......Page 257
Case Study 47: Maintaining Performance Measurement Baseline Data, from National Airspace System, GAO-03-343......Page 293
1.Best Practices Checklist: The Estimate......Page 63
3.Best Practices Checklist: Cost Assessment Team......Page 74
4.Best Practices Checklist: Technical Baseline Description......Page 82
5.Best Practices Checklist: Work Breakdown Structure......Page 96
6.Best Practices Checklist: Ground Rules and Assumptions......Page 106
7.Best Practices Checklist: Data......Page 123
8.Best Practices Checklist: Developing a Point Estimate......Page 141
9.Best Practices Checklist: Estimating Software Costs......Page 163
10.Best Practices Checklist: Sensitivity Analysis......Page 169
11.Best Practices Checklist: Cost Risk and Uncertainty......Page 194
12.Best Practices Checklist: Validating the Estimate......Page 207
13.Best Practices Checklist: Documenting the Estimate......Page 214
14.Best Practices Checklist: Presenting the Estimate to Management......Page 216
15.Best Practices Checklist: Managing Program Costs: Planning......Page 258
17.Best Practices Checklist: Managing Program Costs: Updating......Page 313