Security Chaos Engineering: Sustaining Resilience in Software and Systems

Preface
        Who Should Read This Book?
        Scope of This Book
            Outline of This Book
        Conventions Used in This Book
        O’Reilly Online Learning
        How to Contact Us
        Acknowledgments
    1. Resilience in Software and Systems
        What Is a Complex System?
            Variety Defines Complex Systems
            Complex Systems Are Adaptive
            The Holistic Nature of Complex Systems
        What Is Failure?
            Acute and Chronic Stressors in Complex Systems
            Surprises in Complex Systems
        What Is Resilience?
            Critical Functionality
            Safety Boundaries (Thresholds)
            Interactions Across Space-Time
            Feedback Loops and Learning Culture
            Flexibility and Openness to Change
        Resilience Is a Verb
        Resilience: Myth Versus Reality
            Myth: Robustness = Resilience
            Myth: We Can and Should Prevent Failure
            Myth: The Security of Each Component Adds Up to Resilience
            Myth: Creating a “Security Culture” Fixes Human Error
        Chapter Takeaways
    2. Systems-Oriented Security
        Mental Models of System Behavior
            How Attackers Exploit Our Mental Models
            Refining Our Mental Models
        Resilience Stress Testing
        The E&E Resilience Assessment Approach
        Evaluation: Tier 1 Assessment
            Mapping Flows to Critical Functionality
            Document Assumptions About Safety Boundaries
            Making Attacker Math Work for You
            Starting the Feedback Flywheel with Decision Trees
            Moving Toward Tier 2: Experimentation
        Experimentation: Tier 2 Assessment
            The Value of Experimental Evidence
            Sustaining Resilience Assessments
        Fail-Safe Versus Safe-to-Fail
            Uncertainty Versus Ambiguity
            Fail-Safe Neglects the Systems Perspective
            The Fragmented World of Fail-Safe
        SCE Versus Security Theater
            What Is Security Theater?
            How Does SCE Differ from Security Theater?
        How to RAVE Your Way to Resilience
            Repeatability: Handling Complexity
            Accessibility: Making Security Easier for Engineers
            Variability: Supporting Evolution
        Chapter Takeaways
    3. Architecting and Designing
        The Effort Investment Portfolio
            Allocating Your Effort Investment Portfolio
            Investing Effort Based on Local Context
        The Four Failure Modes Resulting from System Design
        The Two Key Axes of Resilient Design: Coupling and Complexity
            Designing to Preserve Possibilities
        Coupling in Complex Systems
            The Tight Coupling Trade-Off
            The Dangers of Tight Coupling: Taming the Forest
            Investing in Loose Coupling in Software Systems
            Chaos Experiments Expose Coupling
        Complexity in Complex Systems
            Understanding Complexity: Essential and Accidental
            Complexity and Mental Models
            Introducing Linearity into Our Systems
            Designing for Interactivity: Identity and Access Management
            Navigating Flawed Mental Models
        Chapter Takeaways
    4. Building and Delivering
        Mental Models When Developing Software
        Who Owns Application Security (and Resilience)?
            Lessons We Can Learn from Database Administration Going DevOps
        Decisions on Critical Functionality Before Building
            Defining System Goals and Guidelines on “What to Throw Out the Airlock”
            Code Reviews and Mental Models
            “Boring” Technology Is Resilient Technology
            Standardization of Raw Materials
        Developing and Delivering to Expand Safety Boundaries
            Anticipating Scale and SLOs
            Automating Security Checks via CI/CD
            Standardization of Patterns and Tools
            Dependency Analysis and Prioritizing Vulnerabilities
        Observe System Interactions Across Space-Time (or Make More Linear)
            Configuration as Code
            Fault Injection During Development
            Integration Tests, Load Tests, and Test Theater
            Beware Premature and Improper Abstractions
        Fostering Feedback Loops and Learning During Build and Deliver
            Test Automation
            Documenting Why and When
            Distributed Tracing and Logging
            Refining How Humans Interact with Build and Delivery Practices
        Flexibility and Willingness to Change
            Iteration to Mimic Evolution
            Modularity: Humanity’s Ancient Tool for Resilience
            Feature Flags and Dark Launches
            Preserving Possibilities for Refactoring: Typing
            The Strangler Fig Pattern
        Chapter Takeaways
    5. Operating and Observing
        What Does Operating and Observing Involve?
        Operational Goals in SCE
            The Overlap of SRE and Security
            Measuring Operational Success
            Crafting Success Metrics like Attackers
            The DORA Metrics
            SLOs, SLAs, and Principled Performance Analytics
            Embracing Confidence-Based Security
        Observability for Resilience and Security
            Thresholding to Uncover Safety Boundaries
            Attack Observability
        Scalable Is Safer
            Navigating Scalability
            Automating Away Toil
        Chapter Takeaways
    6. Responding and Recovering
        Responding to Surprises in Complex Systems
            Incident Response and the Effort Investment Portfolio
            Action Bias in Incident Response
            Practicing Response Activities
        Recovering from Surprises
            Blameless Culture
            Blaming Human Error
            Hindsight Bias and Outcome Bias
            The Just-World Hypothesis
            Neutral Practitioner Questions
        Chapter Takeaways
    7. Platform Resilience Engineering
        Production Pressures and How They Influence System Behavior
        What Is Platform Engineering?
        Defining a Vision
        Defining a User Problem
            Local Context Is Critical
            User Personas, Stories, and Journeys
            Understanding How Humans Make Trade-Offs Under Pressure
        Designing a Solution
            The Ice Cream Cone Hierarchy of Security Solutions
            System Design and Redesign to Eliminate Hazards
            Substitute Less Hazardous Methods or Materials
            Incorporate Safety Devices and Guards
            Provide Warning and Awareness Systems
            Apply Administrative Controls Including Guidelines and Training
            Two Paths: The Control Strategy or the Resilience Strategy
            Experimentation and Feedback Loops for Solution Design
        Implementing a Solution
            Fostering Consensus
            Planning for Migration
            Success Metrics
        Chapter Takeaways
    8. Security Chaos Experiments
        Lessons Learned from Early Adopters
            Lesson #1. Start in Nonproduction Environments; You Can Still Learn a Lot
            Lesson #2. Use Past Incidents as a Source of Experiments
            Lesson #3. Publish and Evangelize Experimental Findings
        Setting Experiments Up for Success
        Designing a Hypothesis
        Designing an Experiment
        Experiment Design Specifications
        Conducting Experiments
            Collecting Evidence
        Analyzing and Documenting Evidence
            Capturing Knowledge for Feedback Loops
            Document Experiment Release Notes
        Automating Experiments
        Easing into Chaos: Game Days
        Example Security Chaos Experiments
            Security Chaos Experiments for Production Infrastructure
            Security Chaos Experiments for Build Pipelines
            Security Chaos Experiments in Cloud Native Environments
            Security Chaos Experiments in Windows Environments
        Chapter Takeaways
    9. Security Chaos Engineering in the Wild
        Experience Report: The Existence of Order Through Chaos (UnitedHealth Group)
            The Story of ChaoSlingr
            Step-by-Step Example: PortSlingr
        Experience Report: A Quest for Stronger Reliability (Verizon)
            The Bigger They Are…
            All Hands on Deck Means No Hands on the Helm
            Assert Your Hypothesis
            Reliability Experiments
            Cost Experiments
            Performance Experiments
            Risk Experiments
            More Traditionally Known Experiments
            Changing the Paradigm to Continuous
            Lessons Learned
        Experience Report: Security Monitoring (OpenDoor)
        Experience Report: Applied Security (Cardinal Health)
            Building the SCE Culture
            The Mission of Applied Security
            The Method: Continuous Verification and Validation (CVV)
            The CVV Process Includes Four Steps
        Experience Report: Balancing Reliability and Security via SCE (Accenture Global)
            Our Roadmap to SCE Enterprise Capability
            Our Process for Adoption
        Experience Report: Cyber Chaos Engineering (Capital One)
            What Does All This Have to Do with SCE?
            What Is Secure Today May Not Be Secure Tomorrow
            How We Started
            How We Did This in Ye Olden Days
            Things I’ve Learned Along the Way
            A Reduction of Guesswork
            Driving Value
            Conclusion
        Chapter Takeaways
    Index
    About the Authors