Continuous Deployment: Enable Faster Feedback, Safer Releases, and More Reliable Software

Foreword
Preface
   Who This Book Is For
   What You Need to Read This Book
   What This Book Will Cover
   Conventions Used in This Book
   O’Reilly Online Learning
   How to Contact Us
   Acknowledgments
I. Continuous Deployment
1. Continuous Deployment
   Deploying Every Few Months or Years
   Deploying Every Few Days
   Deploying Continuously
   eXtreme Programming
      If It Hurts, Do It More Often
         Smaller, less-painful batches
         Incentive for automation
   DevOps
      The Barrier Between Dev and Ops
      Joining Dev and Ops
      Automation, Automation, Automation
   Continuous Integration
   Continuous Delivery
   A Final Gate to Production
      One Step Further: Continuous Deployment
      Implementation
   Implications
   Is It Dangerous?
   Summary
2. Benefits
   One-Piece Flow and Lean Manufacturing
      Origins of Lean Manufacturing
         Batch production
         One-piece flow
            Reducing batch sizes
            Lowering transaction costs
      Lean and Software Development
         Inventory in software
         Batches in software
         Lowering transaction cost in software
            One commit = one artifact with continuous integration
            One commit = one preproduction deployment with continuous delivery
            One commit = one production deployment with continuous deployment
         Continuous flow ratio
   DORA Metrics
      Throughput Metrics
         Deployment frequency
         Lead time for changes
      Stability Metrics
         Mean time to recover
         Change failure rate
   Shifting Quality Left
      How Continuous Deployments Force a Quality Shift to the Left
      The Effects of Automating Quality Gates
   Summary
3. The Mindset Shift
   Defining a Change Versus Applying a Change
   Hiding Work in Progress
      Version Control Branches
      Execution Branches
         Feature toggles
            Implementation
            Benefits
            What about static feature toggles?
         Expand and contract
            Implementation
            Benefits
         Feature toggles versus expand and contract by type of change
         A conclusion on hiding work in progress
   Distributed Systems
      Contracts Between Systems
         Formal contracts
         Informal contracts
   Contracts Between Paths to Production
      Adding New Features: When Order Matters
         With a gate to production: Order of deployment
         Without a gate to production: Order of development
         Feature toggles
      Refactoring: When Timing Matters
         With a gate to production: Timing of deployment
         Without a gate to production: Timing of development
            Antipattern: Delaying integration
            Antipattern: Pausing pipelines
         Expand and contract
   A Deployment Is Not a Release
      Releases
      Deployments
      Differences
      Overlap
   End-to-End Delivery Life Cycle
      Without Continuous Deployment
         Doing
         QA
         Done
         Implications
      With Continuous Deployment
         Doing
         QA
         Done
         Implications
   Summary
4. You Must Be This Tall
   Cross-Functional, Autonomous Teams
      Fast Decision Making
      Implementation Autonomy
      Frequent Integration
         Short-lived branches
         Short-lived branches versus feature branches
         Trunk-based development
      Frequent Code Reviews
         Pull requests
         Pair programming
         Psychological safety
      Automated Code Analysis
      Test Automation
         The testing pyramid model
         The Swiss cheese model
         Test-first
         Outside-in
         What about legacy?
      Zero-Downtime Deployments
         Blue/green deployments
         Rolling deployments
         Canary deployments
         Deployment strategies and manual steps
            Antipattern example: Blue/green as a QA tool
            Antipattern example: Partial deployments as a canary release tool
      Observability and Monitoring
         Alerts
         Information versus noise
   Stakeholder Trust
      How Do We Convince the Boss?
      When Are We Ready?
   Summary
5. Challenges
   Systems That Are Sensitive to Deployments
      Interruption of Long-Running Processes
         Mitigation: Switch to messaging or event-based architectures
            Instances whose job is to complete jobs
      Sticky Sessions
         Mitigation: Keep state external
      Invalidation of Client-Side Caches
         Mitigation: Be deliberate about your caching policies
      Scaling Interruptions
         Mitigation: Keep application startup time low
         Mitigation: Introduce more generous scaling policies
         Mitigation: Perform pre-scaling
      A Constant Stream of Cold Instances
         Mitigation: Use external caches
   User-Installed Software
      Desktop Applications
         Mitigation: Introduce self-applying updates
      Mobile Apps
         Mitigation: Shift control to the server side
            Web views
            Progressive web apps
            Server-driven UIs
      Appliances and Other Devices
      Libraries and Frameworks
   Regulated Industries
   Cognitive Load
      Overly Busy Path to Production
         Mitigation: Break your monoliths
         Mitigation: Rethink your teams and domains
      Inattention During Deployments
         Mitigation: Use good alerting
         Mitigation: Keep your pipelines fast
         Mitigation: Shift mindset around development
      Breadth of Knowledge Required
         Mitigation: Introduce comprehensive training programs
      Steep Onboarding Curve
         Mitigation: Use pair programming and mob programming
      Scheduling Development Work
         Mitigation: Core development hours
         Mitigation: Perform team code reviews
   Summary
II. Before Development
6. Slicing Upcoming Work
   Horizontal Versus Vertical Slicing
      Horizontal Slicing
      Vertical Slicing
   With Continuous Deployment
   Effective Vertical Slicing
      MVP
      INVEST
      Small Slices
   Example: The Groceroo Company
      The Feature: “Last-Minute Items”
      The User Interface
      The Admin Interface
      The Problems of Implementing with Horizontal Slicing
         Task 1: Persistence layer
         Task 2: Backend layer
            Create and modify items
            “Add to basket” logic
         Task 3: Frontend layer
            Carousel
            Admin interface
            Feature toggle
      Implementation with Vertical Slicing
         User story 1: Add a simple carousel
         User story 2: Make the carousel configurable
         User story 3: One-click add to basket
         User story 4: Add to basket by different quantities
   Summary
7. Building for Production
   Deployability Requirements
      Hiding with Feature Toggles
         New top-level toggle
         New nested toggle
         Under existing toggle
      Hiding with Expand and Contract
         Alternative system
         Alternative operation
         Alternative field
         Generify field type
      Hiding in Version Control Branch
      Unhidden
      Pausing the Pipeline
   Testability Requirements
      High-Level Automated Tests
      Manual Exploratory Testing
   Observability Requirements
      Maintaining Logs and Metrics
      Maintaining Dashboards and Alerts
   Security Requirements
      New User Input
      Storing New Data
      New Dependencies
      New Infrastructure
   Performance Requirements
      New Network Requests
      Data Size
      Persistence Layer
   A (More) Complete User Story Template
   Example: Adding CFRs to Groceroo User Stories
      User Story 1: Add Simple Carousel
         Deployability requirements
         Testability requirements
         Observability requirements
         Security requirements
         Performance requirements
      User Story 2: Admin Area
         Deployability requirements
         Testability requirements
         Observability requirements
         Security requirements
         Performance requirements
      User Story 3: “Add to Basket” Button
         Deployability requirements
         Testability requirements
         Observability requirements
         Security requirements
         Performance requirements
      User Story 4: Quantity Selector
         Deployability
         Testability
         Observability
         Security
         Performance
   Summary
III. During Development
8. Adding New Features
   Our User Story
   The Groceroo Application
      Current State
      Target State
         Frontend
         Backend
         Persistence
      How Do We Get There?
         Feature toggles
         Multiple layers: Outside in
      Implementing with a Feature Toggle
         Deployment 1: Introducing the toggle
         Deployment 2: Adding code in the UI layer
         Deployment 3: Adding code in the backend layer
         Deployment 4: Adding code in the persistence layer
         Release
         Deployment 5: Toggle cleanup
   Summary
9. Refactoring Live Features
   Our Task
   The Product Identifier System
      The Problem
      The Solution
   Current State
      Frontend
      Backend
      Persistence
   Target State
      Frontend
      Backend
      Persistence
   How Do We Get There?
      Expand and Contract
      Multiple Layers: Inside Out
   Implementing with Expand and Contract
      Outer Expand Phase: The Product Table
         Deployment 1: Expand the product table
      Migrating the POST /product Endpoint
         Deployment 2: Expanding the POST /product endpoint
         Deployment 3: Migrating the POST /product endpoint
         Deployment 4: Contracting the POST /product endpoint
      Migrating the GET /product Endpoint
         Deployment 5: Expanding the GET /product endpoint
         Deployment 6: Migrating the GET /product endpoint
         Deployment 7: Contracting the GET /product endpoint
      Migrating the Basket Table
         Deployment 8: Expanding the basket table
         Deployment 9: Expanding the POST /basket endpoint
         Deployment 10: Migrating the POST /basket endpoint
         Deployment 11: Contracting the POST /basket endpoint
         Deployment 12: Contracting the basket table
      Outer Contract Phase: Cleaning Up the Product Table
         Deployment 13: Contracting the product table
   Summary
10. Data and Data Loss
   Our Task
      The Problem
      The Solution
   Current State
      Persistence
      Backend
   Target State
      Persistence
      Backend
   How Do We Get There?
      Failure Mode: Simultaneous Change
      Failure Mode: Simple Expand and Contract
      Solution: Temporary Database Trigger
      Solution: Double-Write
         Step 1: Expand the database column
         Step 2: Double-write to both columns
         Step 3: Synchronize the data
         Step 4: Migrate write and read the columns
         Step 5: Contract the columns
      Solution: Double-Read
         Step 1: Expand the database column
         Step 2: Double-read and migrate write to both columns
         Step 3: Synchronize the data
         Step 4: Migrate the read
         Step 5: Contract the old column
   Implementing with Double-Write
      Deployment 1: Expand the Database Column
      Deployment 2: Double-Write to Both Columns
      Deployment 3: Synchronize the Data
      Deployment 4: Migrate Write and Read to Both Columns
      Deployment 5: Contract the Columns
   Implementing with Double-Read
      Deployment 1: Expand the Database Column
      Deployment 2: Double-Read and Migrate Write the Columns
      Deployment 3: Synchronize the Data
      Deployment 4: Migrate Read the Column
      Deployment 5: Contract the Columns
   NoSQL
      Migrate on Read
         Migrate on read forever
         Migrate on read and convert on write
      Custom Batch Update
      Conclusion on NoSQL
   Summary
IV. After Development
11. Testing in Production
   Why You Should Test in Production
      Data Volume Accuracy
      Data Shape Accuracy
      Realistic Request Patterns
      Realistic Incoming Traffic Volume
      Realistic Outgoing Traffic Volume
      Realistic Size and Number of Servers
      Realistic Application Configuration
      Realistic Network Configuration
      Real Version of Other Teams’ Services
      Real Version of Third-Party Services
      Lower Cost
      Better Data Hygiene
   How to Test in Production
      Feature Toggle Activation Strategies
         Query parameters
            Where to use
            When to use
         Request headers
            Where to use
            When to use
         Cookies
            Where to use
            When to use
         User identifier
            Where to use
            When to use
         User roles
            Where to use
            When to use
      Challenges
         Managing test data
            Hiding test data
            Preventing test data leakages
         Debugging frontend code
            The case for source maps in production
   Life After Staging
   Summary
12. Releasing
   Antipattern: Big Bang Releases
      Comparing Before Versus After States
      Comparing Simultaneous States
   Antipattern: Partial Releases Through Partial Deployments
   Using Feature Flags for Releases
      Coordinating Feature Flag Releases in Distributed Systems
      Antipattern: Independent Flag State for Each Service
      Propagating Flag State Down the Call Chain
      Centralized Feature Flag State
   Canary Releases
      By Traffic Percentage
      By Device
         Per-device toggle
         Backend-controlled toggle
      By Country
      By User Segment
   A/B Testing
      Analytics
      Experiment Best Practices
         Test one variable at a time
         Avoid test parallelism
         Have a sufficient sample size
      Schedule Delivery Around Experiments
      Types of A/B Tests
         Tests with two variants
         Split or redirect tests
         Multivariate tests
            Full factorial
            Fractional factorial
   Do You Need a Canary Release or an A/B Test?
   Summary
Conclusion and Next Steps
V. Case Studies
Case Study: AutoScout24
   AutoScout24’s Context
   Adoption of Continuous Deployment at AutoScout24
      Overcoming Organizational Obstacles
      Overcoming Technical Obstacles
   AutoScout24’s Implementation of Continuous Deployment
      How AutoScout24 Makes Continuous Deployment Safe
         Code review process
         Test automation
         Zero-downtime deployments
         Observability
   How AutoScout24 Supports Junior Engineers
Case Study: OTTO
   OTTO’s Context
   Adoption of Continuous Deployment at OTTO
      Working with Organizational Barriers
      Working with Technical Barriers
   OTTO’s Implementation of Continuous Deployment
      OTTO’s Path to Production
      How OTTO Makes Continuous Deployment Safe
         Team culture
         Resilience patterns
         Test automation
         Deployment strategy
         The four-eyes principle
      How OTTO Supports Junior Engineers with Continuous Deployment
      Links and Resources
Case Study: N26
   N26’s Context
   Adoption of Continuous Deployment at N26
      Working Within Industry Regulations
      Working with Technical Barriers
   N26’s Implementation of Continuous Deployment
      How N26 Makes Continuous Deployment Safe
         Test automation
         Deployment strategy
         Four-eyes principle
         Observability
      How N26 Supports Junior Engineers
   Links and Resources
Case Study: ClimatePartner
   ClimatePartner’s Context
   Adoption of Continuous Deployment at ClimatePartner
   ClimatePartner’s Implementation of Continuous Deployment
      How ClimatePartner Makes Continuous Deployment Safe
         Test automation
         Zero downtime
         Feature toggles
         Observability
         Four-eyes principle
         Testing in production
      How ClimatePartner Supports Junior Engineers with Continuous Deployment
Case Study: Motability Operations
   Motability Operations’ Context
   Adoption of Continuous Deployment at Motability Operations
      Working with Organizational Barriers
      Working with technical barriers
   Motability Operations’ Implementation of Continuous Deployment
      How Motability Makes Continuous Deployment Safe
         Test automation
         Zero-downtime deployments
         Continuous code reviews
         Observability
         Feature flags
         Encouraging contributions
      How Motability Operations Supports Teams
         Junior engineers
         Hiring
         Onboarding
Case Study: REA Group
   REA’s Context
   Adoption of Continuous Deployment at REA
      Overcoming organizational barriers
         Overcoming technical barriers
   REA’s Implementation of Continuous Deployment
      How REA Makes Continuous Deployment Safe
         Test automation
         Zero-downtime deployments
         Code reviews
         Observability
      How REA Supports Junior Engineers with Continuous Deployment
Case Study: Maze
   Maze’s Context
   Adoption of Continuous Deployment at Maze
      Overcoming Organizational Challenges
      Overcoming Technical Challenges
   Maze’s Implementation of Continuous Deployment
      How Maze Makes Continuous Deployment Safe
      How Maze Supports Junior Engineers with Continuous Deployment
Case Study: TravelPerk
   TravelPerk’s Context
   Adoption of Continuous Deployment at TravelPerk
   TravelPerk’s Implementation of Continuous Deployment
      How TravelPerk Makes Continuous Deployment Safe
         Hiding with work in progress
         Code reviews
         Automated tests
         Zero downtime
         Observability
      How TravelPerk Onboards Engineers to Continuous Deployment
Index