The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

Gene Kim......Page 6
Jez Humble......Page 7
Patrick Debois......Page 8
John Willis......Page 9
SPREADING THE AHA! MOMENT......Page 13
Foreword......Page 15
An Introduction to The DevOps Handbook......Page 17
THE PROBLEM: SOMETHING IN YOUR ORGANIZATION MUST NEED IMPROVEMENT (OR YOU WOULDN’T BE READING THIS BOOK)......Page 21
THE CORE, CHRONIC CONFLICT......Page 22
DOWNWARD SPIRAL IN THREE ACTS......Page 23
WHY DOES THIS DOWNWARD SPIRAL HAPPEN EVERYWHERE?......Page 25
THE COSTS: HUMAN AND ECONOMIC......Page 26
THE ETHICS OF DEVOPS: THERE IS A BETTER WAY......Page 27
BREAKING THE DOWNWARD SPIRAL WITH DEVOPS......Page 28
THE BUSINESS VALUE OF DEVOPS......Page 31
DEVOPS HELPS SCALE DEVELOPER PRODUCTIVITY......Page 33
THE UNIVERSALITY OF THE SOLUTION......Page 34
THE DEVOPS HANDBOOK: AN ESSENTIAL GUIDE......Page 36
A BRIEF HISTORY......Page 40
THE LEAN MOVEMENT......Page 41
AGILE INFRASTRUCTURE AND VELOCITY MOVEMENT......Page 42
TOYOTA KATA......Page 43
THE MANUFACTURING VALUE STREAM......Page 45
THE TECHNOLOGY VALUE STREAM......Page 46
FOCUS ON DEPLOYMENT LEAD TIME......Page 47
The Common Scenario: Deployment Lead Times Requiring Months......Page 48
Our DevOps Ideal: Deployment Lead Times of Minutes......Page 49
THE THREE WAYS: THE PRINCIPLES UNDERPINNING DEVOPS......Page 50
CONCLUSION......Page 53
MAKE OUR WORK VISIBLE......Page 54
LIMIT WORK IN PROCESS (WIP)......Page 57
REDUCE BATCH SIZES......Page 58
REDUCE THE NUMBER OF HANDOFFS......Page 62
CONTINUALLY IDENTIFY AND ELEVATE OUR CONSTRAINTS......Page 63
ELIMINATE HARDSHIPS AND WASTE IN THE VALUE STREAM......Page 65
CONCLUSION......Page 67
3 The Second Way: The Principles of Feedback......Page 69
WORKING SAFELY WITHIN COMPLEX SYSTEMS......Page 70
SEE PROBLEMS AS THEY OCCUR......Page 71
SWARM AND SOLVE PROBLEMS TO BUILD NEW KNOWLEDGE......Page 73
KEEP PUSHING QUALITY CLOSER TO THE SOURCE......Page 76
ENABLE OPTIMIZING FOR DOWNSTREAM WORK CENTERS......Page 78
CONCLUSION......Page 79
4 The Third Way: The Principles of Continual Learning and Experimentation......Page 80
ENABLING ORGANIZATIONAL LEARNING AND A SAFETY CULTURE......Page 81
INSTITUTIONALIZE THE IMPROVEMENT OF DAILY WORK......Page 84
TRANSFORM LOCAL DISCOVERIES INTO GLOBAL IMPROVEMENTS......Page 86
INJECT RESILIENCE PATTERNS INTO OUR DAILY WORK......Page 87
LEADERS REINFORCE A LEARNING CULTURE......Page 89
CONCLUSION......Page 91
PART I CONCLUSION......Page 93
Introduction......Page 95
5 Selecting Which Value Stream to Start With......Page 97
GREENFIELD VS. BROWNFIELD SERVICES......Page 101
CONSIDER BOTH SYSTEMS OF RECORD AND SYSTEMS OF ENGAGEMENT......Page 104
START WITH THE MOST SYMPATHETIC AND INNOVATIVE GROUPS......Page 105
EXPANDING DEVOPS ACROSS OUR ORGANIZATION......Page 107
CONCLUSION......Page 108
6 Understanding the Work in Our Value Stream, Making it Visible, and Expanding it Across the Organization......Page 110
IDENTIFYING THE TEAMS SUPPORTING OUR VALUE STREAM......Page 112
CREATE A VALUE STREAM MAP TO SEE THE WORK......Page 113
CREATING A DEDICATED TRANSFORMATION TEAM......Page 116
AGREE ON A SHARED GOAL......Page 118
KEEP OUR IMPROVEMENT PLANNING HORIZONS SHORT......Page 119
RESERVE 20% OF CYCLES FOR NON-FUNCTIONAL REQUIREMENTS AND REDUCING TECHNICAL DEBT......Page 120
USE TOOLS TO REINFORCE DESIRED BEHAVIOR......Page 126
CONCLUSION......Page 128
7 How to Design Our Organization and Architecture with Conway’s Law in Mind......Page 129
ORGANIZATIONAL ARCHETYPES......Page 133
PROBLEMS OFTEN CAUSED BY OVERLY FUNCTIONAL ORIENTATION (“OPTIMIZING FOR COST”)......Page 134
ENABLE MARKET-ORIENTED TEAMS (“OPTIMIZING FOR SPEED”)......Page 135
MAKING FUNCTIONAL ORIENTATION WORK......Page 136
TESTING, OPERATIONS, AND SECURITY AS EVERYONE’S JOB, EVERY DAY......Page 138
ENABLE EVERY TEAM MEMBER TO BE A GENERALIST......Page 139
FUND NOT PROJECTS, BUT SERVICES AND PRODUCTS......Page 142
DESIGN TEAM BOUNDARIES IN ACCORDANCE WITH CONWAY’S LAW......Page 143
CREATE LOOSELY-COUPLED ARCHITECTURES TO ENABLE DEVELOPER PRODUCTIVITY AND SAFETY......Page 144
KEEP TEAM SIZES SMALL (THE “TWO-PIZZA TEAM” RULE)......Page 145
CONCLUSION......Page 150
8 How to Get Great Outcomes by Integrating Operations into the Daily Work of Development......Page 152
CREATE SHARED SERVICES TO INCREASE DEVELOPER PRODUCTIVITY......Page 154
EMBED OPS ENGINEERS INTO OUR SERVICE TEAMS......Page 157
ASSIGN AN OPS LIAISON TO EACH SERVICE TEAM......Page 159
INTEGRATE OPS INTO DEV RITUALS......Page 160
INVITE OPS TO OUR DEV RETROSPECTIVES......Page 161
MAKE RELEVANT OPS WORK VISIBLE ON SHARED KANBAN BOARDS......Page 163
CONCLUSION......Page 164
PART II CONCLUSION......Page 165
PART III—THE FIRST WAY: THE TECHNICAL PRACTICES OF FLOW......Page 166
9 Create the Foundations of Our Deployment Pipeline......Page 169
ENABLE ON DEMAND CREATION OF DEV, TEST, AND PRODUCTION ENVIRONMENTS......Page 171
CREATE OUR SINGLE REPOSITORY OF TRUTH FOR THE ENTIRE SYSTEM......Page 174
MAKE INFRASTRUCTURE EASIER TO REBUILD THAN TO REPAIR......Page 177
MODIFY OUR DEFINITION OF DEVELOPMENT “DONE” TO INCLUDE RUNNING IN PRODUCTION-LIKE ENVIRONMENTS......Page 179
CONCLUSION......Page 181
10 Enable Fast and Reliable Automated Testing......Page 183
CONTINUOUSLY BUILD, TEST, AND INTEGRATE OUR CODE AND ENVIRONMENTS......Page 187
BUILD A FAST AND RELIABLE AUTOMATED VALIDATION TEST SUITE......Page 190
CATCH ERRORS AS EARLY IN OUR AUTOMATED TESTING AS POSSIBLE......Page 193
ENSURE TESTS RUN QUICKLY (IN PARALLEL, IF NECESSARY)......Page 195
WRITE OUR AUTOMATED TESTS BEFORE WE WRITE THE CODE (“TEST-DRIVEN DEVELOPMENT”)......Page 196
AUTOMATE AS MANY OF OUR MANUAL TESTS AS POSSIBLE......Page 197
INTEGRATE PERFORMANCE TESTING INTO OUR TEST SUITE......Page 199
INTEGRATE NON-FUNCTIONAL REQUIREMENTS TESTING INTO OUR TEST SUITE......Page 200
PULL OUR ANDON CORD WHEN THE DEPLOYMENT PIPELINE BREAKS......Page 201
WHY WE NEED TO PULL THE ANDON CORD......Page 203
CONCLUSION......Page 204
11 Enable and Practice Continuous Integration......Page 206
SMALL BATCH DEVELOPMENT AND WHAT HAPPENS WHEN WE COMMIT CODE TO TRUNK INFREQUENTLY......Page 210
ADOPT TRUNK-BASED DEVELOPMENT PRACTICES......Page 212
CONCLUSION......Page 217
12 Automate and Enable Low-Risk Releases......Page 218
AUTOMATE OUR DEPLOYMENT PROCESS......Page 221
ENABLE AUTOMATED SELF-SERVICE DEPLOYMENTS......Page 226
INTEGRATE CODE DEPLOYMENT INTO THE DEPLOYMENT PIPELINE......Page 228
DECOUPLE DEPLOYMENTS FROM RELEASES......Page 232
The Blue-Green Deployment Pattern......Page 235
Dealing with Database Changes......Page 236
The Canary and Cluster Immune System Release Patterns......Page 239
Implement Feature Toggles......Page 241
Perform Dark Launches......Page 243
SURVEY OF CONTINUOUS DELIVERY AND CONTINUOUS DEPLOYMENT IN PRACTICE......Page 246
CONCLUSION......Page 248
13 Architect for Low-Risk Releases......Page 250
AN ARCHITECTURE THAT ENABLES PRODUCTIVITY, TESTABILITY, AND SAFETY......Page 252
ARCHITECTURAL ARCHETYPES: MONOLITHS VS. MICROSERVICES......Page 254
USE THE STRANGLER APPLICATION PATTERN TO SAFELY EVOLVE OUR ENTERPRISE ARCHITECTURE......Page 258
CONCLUSION......Page 263
PART III CONCLUSION......Page 265
Introduction......Page 267
14 Create Telemetry to Enable Seeing and Solving Problems......Page 269
CREATE OUR CENTRALIZED TELEMETRY INFRASTRUCTURE......Page 273
CREATE APPLICATION LOGGING TELEMETRY THAT HELPS PRODUCTION......Page 277
USE TELEMETRY TO GUIDE PROBLEM SOLVING......Page 280
ENABLE CREATION OF PRODUCTION METRICS AS PART OF DAILY WORK......Page 282
CREATE SELF-SERVICE ACCESS TO TELEMETRY AND INFORMATION RADIATORS......Page 284
FIND AND FILL ANY TELEMETRY GAPS......Page 288
APPLICATION AND BUSINESS METRICS......Page 289
INFRASTRUCTURE METRICS......Page 292
OVERLAYING OTHER RELEVANT INFORMATION ONTO OUR METRICS......Page 293
CONCLUSION......Page 294
15 Analyze Telemetry to Better Anticipate Problems and Achieve Goals......Page 296
USE MEANS AND STANDARD DEVIATIONS TO DETECT POTENTIAL PROBLEMS......Page 298
INSTRUMENT AND ALERT ON UNDESIRED OUTCOMES......Page 300
PROBLEMS THAT ARISE WHEN OUR TELEMETRY DATA HAS NON-GAUSSIAN DISTRIBUTION......Page 301
USING ANOMALY DETECTION TECHNIQUES......Page 306
CONCLUSION......Page 311
16 Enable Feedback So Development and Operations Can Safely Deploy Code......Page 312
USE TELEMETRY TO MAKE DEPLOYMENTS SAFER......Page 315
DEV SHARES PAGER ROTATION DUTIES WITH OPS......Page 317
HAVE DEVELOPERS FOLLOW WORK DOWNSTREAM......Page 318
HAVE DEVELOPERS INITIALLY SELF-MANAGE THEIR PRODUCTION SERVICE......Page 320
CONCLUSION......Page 328
17 Integrate Hypothesis-Driven Development and A/B Testing into Our Daily Work......Page 330
A BRIEF HISTORY OF A/B TESTING......Page 332
INTEGRATING A/B TESTING INTO OUR FEATURE TESTING......Page 333
INTEGRATE A/B TESTING INTO OUR RELEASE......Page 335
INTEGRATING A/B TESTING INTO OUR FEATURE PLANNING......Page 336
CONCLUSION......Page 339
18 Create Review and Coordination Processes to Increase Quality of Our Current Work......Page 341
THE DANGERS OF CHANGE APPROVAL PROCESSES......Page 343
POTENTIAL DANGERS OF “OVERLY CONTROLLING CHANGES”......Page 345
ENABLE COORDINATION AND SCHEDULING OF CHANGES......Page 347
ENABLE PEER REVIEW OF CHANGES......Page 348
ENABLE PAIR PROGRAMMING TO IMPROVE ALL OUR CHANGES......Page 353
EVALUATING THE EFFECTIVENESS OF PULL REQUEST PROCESSES......Page 357
FEARLESSLY CUT BUREAUCRATIC PROCESSES......Page 359
CONCLUSION......Page 361
PART IV CONCLUSION......Page 362
Introduction......Page 364
19 Enable and Inject Learning into Daily Work......Page 366
ESTABLISH A JUST, LEARNING CULTURE......Page 368
SCHEDULE BLAMELESS POST-MORTEM MEETINGS AFTER ACCIDENTS OCCUR......Page 370
PUBLISH OUR POST-MORTEMS AS WIDELY AS POSSIBLE......Page 373
DECREASE INCIDENT TOLERANCES TO FIND EVER-WEAKER FAILURE SIGNALS......Page 375
REDEFINE FAILURE AND ENCOURAGE CALCULATED RISK-TAKING......Page 377
INJECT PRODUCTION FAILURES TO ENABLE RESILIENCE AND LEARNING......Page 378
INSTITUTE GAME DAYS TO REHEARSE FAILURES......Page 380
CONCLUSION......Page 383
USE CHAT ROOMS AND CHAT BOTS TO AUTOMATE AND CAPTURE ORGANIZATIONAL KNOWLEDGE......Page 385
AUTOMATE STANDARDIZED PROCESSES IN SOFTWARE FOR RE-USE......Page 388
CREATE A SINGLE, SHARED SOURCE CODE REPOSITORY FOR OUR ENTIRE ORGANIZATION......Page 389
SPREAD KNOWLEDGE BY USING AUTOMATED TESTS AS DOCUMENTATION AND COMMUNITIES OF PRACTICE......Page 392
DESIGN FOR OPERATIONS THROUGH CODIFIED NON-FUNCTIONAL REQUIREMENTS......Page 393
BUILD REUSABLE OPERATIONS USER STORIES INTO DEVELOPMENT......Page 395
ENSURE TECHNOLOGY CHOICES HELP ACHIEVE ORGANIZATIONAL GOALS......Page 396
CONCLUSION......Page 399
21 Reserve Time to Create Organizational Learning and Improvement......Page 400
INSTITUTIONALIZE RITUALS TO PAY DOWN TECHNICAL DEBT......Page 402
ENABLE EVERYONE TO TEACH AND LEARN......Page 405
SHARE YOUR EXPERIENCES FROM DEVOPS CONFERENCES......Page 406
CREATE INTERNAL CONSULTING AND COACHES TO SPREAD PRACTICES......Page 409
CONCLUSION......Page 412
CONCLUSION TO PART V......Page 413
22 Information Security as Everyone’s Job, Every Day......Page 415
INTEGRATE SECURITY INTO DEVELOPMENT ITERATION DEMONSTRATIONS......Page 416
INTEGRATE SECURITY INTO DEFECT TRACKING AND POST-MORTEMS......Page 417
INTEGRATE PREVENTIVE SECURITY CONTROLS INTO SHARED SOURCE CODE REPOSITORIES AND SHARED SERVICES......Page 418
INTEGRATE SECURITY INTO OUR DEPLOYMENT PIPELINE......Page 420
ENSURE SECURITY OF THE APPLICATION......Page 422
ENSURE SECURITY OF OUR SOFTWARE SUPPLY CHAIN......Page 428
ENSURE SECURITY OF THE ENVIRONMENT......Page 430
INTEGRATE INFORMATION SECURITY INTO PRODUCTION TELEMETRY......Page 432
CREATING SECURITY TELEMETRY IN OUR ENVIRONMENT......Page 434
PROTECT OUR DEPLOYMENT PIPELINE......Page 438
CONCLUSION......Page 439
INTEGRATE SECURITY AND COMPLIANCE INTO CHANGE APPROVAL PROCESSES......Page 441
RE-CATEGORIZE THE MAJORITY OF OUR LOWER RISK CHANGES AS STANDARD CHANGES......Page 443
WHAT TO DO WHEN CHANGES ARE CATEGORIZED AS NORMAL CHANGES......Page 444
REDUCE RELIANCE ON SEPARATION OF DUTY......Page 448
ENSURE DOCUMENTATION AND PROOF FOR AUDITORS AND COMPLIANCE OFFICERS......Page 452
CONCLUSION......Page 457
PART VI CONCLUSION......Page 458
Conclusion to the DevOps Handbook......Page 459
THE LEAN MOVEMENT......Page 463
THE AGILE INFRASTRUCTURE MOVEMENT......Page 464
THE TOYOTA KATA MOVEMENT......Page 465
THE RUGGED COMPUTING MOVEMENT......Page 466
APPENDIX 2 THEORY OF CONSTRAINTS AND CORE, CHRONIC CONFLICTS......Page 467
APPENDIX 3 TABULAR FORM OF DOWNWARD SPIRAL......Page 468
APPENDIX 4 THE DANGERS OF HANDOFFS AND QUEUES......Page 469
APPENDIX 5 MYTHS OF INDUSTRIAL SAFETY......Page 472
APPENDIX 6 THE TOYOTA ANDON CORD......Page 473
APPENDIX 7 COTS SOFTWARE......Page 474
APPENDIX 8 POST-MORTEM MEETINGS......Page 475
APPENDIX 9 THE SIMIAN ARMY......Page 477
APPENDIX 10 TRANSPARENT UPTIME......Page 479
Additional Resources......Page 480
INTRODUCTION......Page 484
PART I INTRODUCTION......Page 486
CHAPTER 2......Page 487
CHAPTER 3......Page 488
CHAPTER 4......Page 490
CHAPTER 5......Page 491
CHAPTER 6......Page 494
CHAPTER 7......Page 496
CHAPTER 8......Page 500
CHAPTER 9......Page 501
CHAPTER 10......Page 502
CHAPTER 11......Page 505
CHAPTER 12......Page 506
CHAPTER 13......Page 510
CHAPTER 14......Page 512
CHAPTER 15......Page 516
CHAPTER 16......Page 518
CHAPTER 17......Page 519
CHAPTER 18......Page 521
CHAPTER 19......Page 524
CHAPTER 20......Page 529
CHAPTER 21......Page 531
CHAPTER 22......Page 534
CHAPTER 23......Page 537
CONCLUSION......Page 539
APPENDIX......Page 540
Symbols Numbers A B C D E F G H I J K L M N O P Q R S T U V W Y Z......Page 542
John Willis......Page 595
Gene Kim......Page 596
JOHN WILLIS......Page 599