Enabling Microservice Success: Managing Technical, Organizational, and Cultural Challenges

Foreword
Preface
   Why I Wrote This Book
   Who Should Read This Book
   Navigating This Book
      Part I: Context
      Part II: Organizational Structure and Culture
      Part III: Building and Operating
      Appendixes
      Case Studies
   Conventions Used in This Book
   O’Reilly Online Learning
   How to Contact Us
   Acknowledgments
I. Context
1. Understanding Microservices
   Defining the Microservices Architectural Style
      A Suite of Services
      Each Running in Its Own Process
      Communicating with Lightweight Mechanisms
      Built Around Business Capabilities
      Independently Deployable
      “Small”
      With a Bare Minimum of Centralized Management
      Heterogeneous
   Forerunners and Alternatives
      The Monolith
      Modular Monoliths
      Service-Oriented Architecture
   The Microservices Ecosystem
      Infrastructure as Code
      Continuous Delivery
      The Public Cloud
      New Deployment Options
         Containers
         Orchestration
         Platform-as-a-service options
         Serverless
         Making your choice
      DevOps
      Observability
   Advantages of Microservices
      Independently Scalable
      Robust
      Easy to Release Small Changes Frequently
      Support Flexible Technology Choices
   Challenges of Microservices
      Latency
      Estate Complexity
      Operational Complexity
      Data Consistency
      Security
      Finding the Right Level of Granularity
      Handling Change
      Require Organizational Change
      Change the Developer Experience
   In Summary
2. Effective Software Delivery
   Regularly Delivering Business Value
      High Deployment Frequency
      Short Lead Time for Changes
      Running Experiments
      Separating Deploying Code from Releasing Functionality
      Handling Work That Goes Across Team Boundaries
   Adapting to Changing Priorities
   Maintaining Appropriate Service Levels
      When a Release Goes Wrong
      Knowing When Something Important Is Broken
      Restore Some Level of Service Quickly
      Avoid Failure Cascades
   Spending Most of Your Time on Meaningful Work
   Not Having to Start Again
   Keeping Risk at an Acceptable Level
   How Microservices Measure Up
   In Summary
3. Are Microservices Right for You?
   Reasons to Choose Microservices
      Scaling the Organization
      Developer Experience
      Separating Out Areas with Compliance and Security Requirements
      Scaling for Load
      Increasing Robustness
      Increasing Flexibility
   Conditions for Success
      Domain Understanding
      Products Not Projects
      Leadership Support
      Teams That Want Autonomy
      Processes That Enable Autonomy
      Technical Maturity
   Managing Change
   Sticking with a Monolithic Architecture
      Enable Zero-Downtime Deployments
      Build a Modular Monolith
   Everything Is Distributed Now
      The Rise of Cloud Native
      SaaS Makes Sense
   Recommendations
      Starting from Scratch
      Replacing an Existing Monolith
      Measuring Success
   In Summary
II. Organizational Structure and Culture
4. Conway’s Law and Finding the Right Boundaries
   Conway’s Law
   The Inverse Conway Maneuver
   Possible Boundaries
      Business Domains
         Data
      Locations
      Technologies
      Compliance
      Tolerance for Failure
      Frequency of Changes
      Recommendations
   Identifying When Boundaries Are Wrong
   In Summary
5. Building Effective Teams
   Organizational Culture
      Open
      Learning
      Empowering
      Optimized for Change
      The Westrum Model
   Effective Teams
      Motivated through Autonomy, Mastery, and Purpose
      Aligned to Business Domain
      Appropriately Sized
      Cross-Functional and T-shaped
      Strong Ownership
      Long Lived
      Sustainable Cognitive Load
      High Trust and High Psychological Safety
      Part of a Group
   Optimizing for Flow
      Stream-Aligned
      Enabling
      Complicated Subsystem
      Platform
   In Summary
6. Enabling Autonomy
   What Is Autonomy?
      Why Does Autonomy Matter?
      Limits to Autonomy
   The Right Amount of Communication
   Interaction Styles
      Collaboration
      X-as-a-Service
      Facilitating
   Ways of Working That Support Autonomy
      Aligning on Outcomes
      Light Touch Governance
      Trust but Verify
      Agreeing and Aligning on Technology
      The Role of the Individual Contributor
      Minimum Viable Competencies
      Making Space for Learning
   Responsibilities of Autonomous Teams
      Active Ownership
      Communication and Cooperation
      Compliance with Standards
      Maintaining a Team Page
   In Summary
7. Engineering Enablement and Paving the Road
   What’s in a Name?
   Building a Platform
      Platform Services
      Organization-Level Concerns
         Infrastructure efficiency and costs
         Security and compliance
         Organization-wide changes
      Building the Thinnest Viable Platform
      Build for the Needs of the Majority
      Platform as a Product
   Beyond the Platform
      Vendor Engineering
      APIs, Templates, Libraries, and Examples
      A Service Catalog
      Insights
   Paving the Road
      What Capabilities to Include
      Make It Optional
      Keep It Small
      How to Go Off Road
      Bringing the Treasure Back
      Internal Developer Portals
   Building a Platform People Actually Use
      Making Sure What You Build Meets a Need
      Market It
      Look for Signs You Are Getting It Wrong
   Principles for Building a Paved Road
      Optional
      Provides Value
      Self-Service
      Owned and Supported
      Easy to Use
      Guides People to Do the Right Thing
      Composable and Extendable
   Measuring Impact
   When to Invest in Engineering Enablement
   In Summary
8. Ensuring “You Build It, You Run It”
   Why Microservices Implies DevOps
      Release on Demand
      Work on Operational Features
   Building Things Differently
      Good Runbooks
      Running on Someone Else’s Servers
      Getting Comfortable in Production
   Supporting Your System in Production
      Assign Dedicated In-Hours Ops Support
      Improve Alerts and Documentation
      Identify the Haunted Forests
      Practice
   Out-of-Hours Support
      Allow People to Opt Out
      Formal Rotas Versus Best Endeavors
      Make Sure Calls Are Rare
      Only for Critical Systems
      Provide Support and Guidance
   Incident Management
      Blameless Culture
      Raising an Incident
      Roles to Assign
      During the Incident
      After the Incident
      Learning from Incidents
   In Summary
III. Building and Operating
9. Active Service Ownership
   Responding to the Log4Shell Vulnerability
   A Counter Example: Equifax and a Struts Vulnerability
   Ownership During Active Development
      Strong Ownership
      Weak Ownership
      Collective Ownership
   Once a Service Is Feature Complete
      No Ownership
      Nominal Ownership
      Active Ownership
   What Active Ownership Means
      Code Stewardship
      Upgrades and Patching
      Migrations
      Production Support
      Documentation
   Knowing Your Estate
      Your Own Software
      Dependencies
      Third-Party Software
   What You Need from a Service Catalog
      Graph-Based Model
      API-Driven
      Extensible
      Flexible Schema
      Provides Different Views Across the Estate
   Transferring Ownership
      What Does a Good Transfer Look Like?
      Meeting Quality Expectations
      Operational Handover
      Replacing
   What to Do If You’re Struggling
      Make the Business Case
      Start with Critical Systems
      Make Your Best Guess at Owners
      Deliver Value from the Data
      Aim for Continuous Improvement
      Look for Teams That Are Overwhelmed
      Services Shouldn’t Live Forever
   In Summary
10. Getting Value from Testing
   Why Do We Test?
      Building the Thing Right
      Building the Right Thing
      Picking Up Regressions
      Meeting Quality-of-Service Requirements
   Shifting Testing Left
   What Makes a Good Test?
      Fast and Early Feedback
      Easy to Change
      Finds Real Problems
   Types of Testing
      The Testing Pyramid
      Unit Tests
      Service Tests
      End-to-End Tests
      Contract Tests
      Consistency Tests
      Exploratory Tests
      Cross-Functional Testing
   Testing in Production
      Is It Safe?
      Staging Is Not Production-Like
      Your Customers Can Surprise You
      You Can’t Test for Every Variation
      You Don’t Have to Roll a Change Out to Everyone
         Feature flags
         Canary releases
         A/B testing
      Monitoring as Testing
         Synthetic monitoring
         Coherence testing
   Testing Your Infrastructure
      Chaos Engineering
      Testing Failovers and Restores
   Quality Is About More Than Testing
   What to Do If You’re Struggling
      Not Enough Automated Testing
      Tests That Aren’t Providing Value
   In Summary
11. Governance and Standardization: Finding the Balance
   Why Governance Matters
   Know Your Estate
   What Sort of Information Is Relevant?
   Guardrails and Policies
      Automating Guardrails
      What to Include
      The FT’s Guardrails
         1. Buy vs build
         2. Procurement
         3. Significant technology changes
         4. Creating a record for the service
         5. Security & privacy
         6. Accessibility & browser support
         7. Analytics, logs & metrics
         8. Change & release logging
         9. Healthchecks & monitoring
         10. Runbooks
         11. Service tier & support
         12. Performance
         13. Cost management
         14. Going live
         15. While the service is live
         16. Decommissioning
   Aligning on Guardrails
      Tech Governance Group
         TGG format
         Responsibilities
         What required a technology proposal
      Benefits of the TGG
         Clear and lightweight process
         Easy to know what topics are being discussed
         Record of what people were thinking
         Supports the evolution of your guardrails
   Choosing Technologies
      The Technology Lifecycle
      Save Innovation for Key Business Outcomes
      Use Boring Technology
      Limit the Alternatives
      Be Clear on Where Duplication Is Acceptable
      Expect Things to Change
   Insight Leads to Action
   Governance in Other Organizations
      Governance at Monzo
      Governance at Skyscanner
   What to Do If You’re Struggling
   In Summary
12. Building Resilience In
   What Is Resilience?
      Resilience for Distributed Systems
         Fallacy 1: The network is reliable
         Fallacy 2: Latency is zero
         Fallacy 3: Bandwidth is infinite
         Fallacy 4: The network is secure
         Fallacy 5: Topology doesn’t change
         Fallacy 6: There is one administrator
         Fallacy 7: Transport cost is zero
         Fallacy 8: The network is homogeneous
      Resilience for Microservices
   Understanding Your Service Level Requirements
      Service Level Objectives
         Request duration
         Error rate
         System throughput
         Availability
      Error Budgets
   Building Resilient Services
      Redundancy
      Fast Startup and Graceful Shutdown
      Set Appropriate Timeouts
      Back Off and Retry
      Make Your Requests Idempotent
      Protect Yourself
      Testing Service Resilience
      Make Building Resilient Services Easy
   Building Resilient Systems
      Caching
      Handling Cascading Failures
      Fallback Behavior
      Avoiding Unnecessary Work
      Go Asynchronous
      Failover
      Backup and Restore
      Disaster Recovery
   Building a Resilient Platform
      Resilience to External Issues
      Internal Tooling
         Deployment tooling
         Operational tooling
         Think about data too
   Validating Your Resilience Choices
      Chaos Engineering
      Testing Backup and Restore
      Practice Makes Perfect
      Load Testing
      Learn from Incidents
      One Thing at a Time
   What to Do If You’re Struggling
   In Summary
13. Running Your System in Production
   Operational Challenges of Microservices
      Different Technologies Mean Different Support Knowledge Is Needed
      Ephemeral Infrastructure
      Rapid Change
      Alert Overload
      Complex Systems Run in Degraded Mode
   Building Observability In
      Logging
      Monitoring and Metrics
      Log Aggregation
         Timing issues
         Missing or delayed logs
         Correlation
         Log volume
         Changing vendor
      OpenTelemetry
      Focus on Events
      Distributed Tracing
      Archiving Observability Data
   Building Your Own Tools
   Spotting Issues
      Getting Alerting Right
      Healthchecks
         Aim for a realistic request, rather than a ping
         Return 200 regardless of whether checks pass
         Use healthchecks in dashboards
         Be aware of related costs
      Monitoring Business Outcomes
      Understanding What Normal Looks Like
   Mitigation
   Troubleshooting
      Maintaining Useful Documentation
      Knowing What’s Changed
      Problems with External Systems
      Tooling Characteristics
         Real-time, aggregated data
         Easy to access
         Support investigation
   Learning from Incidents
   What to Do If You’re Struggling
   In Summary
14. Keeping Things Up-to-Date
   Why Is This a Challenge?
   Minimizing the Impact of Change
      Think About the Long Term
      A Reason to Be on the Paved Road
      Choose Managed Services and SaaS Options
      Provide APIs
      Immutable and Ephemeral Infrastructure
      Decommission and Deprecate
   Types of Change
      Emergency Changes
      Minor Planned Changes
      Major Planned Changes
   Responding to Change
      Understand the Landscape
      Define Guiding Policies
   Making a Decision
      Who Gets to Decide?
      Scheduling Work
   Managing Change
      Clarity
      Communication
         Stages of communication
         Don’t break things
         Give lots of notice
         Develop a comms plan
         Give context
      Empathy
         What is a “nudge”?
         Easy
         Attractive
         Social
         Timely
      Execution
   What to Do If You’re Struggling
   In Summary
Afterword
   Why Microservices?
      The Importance of Flow
      Support for Autonomy
      The Rise of Platform Engineering
      Wrapping Up
A. Microservices Assessment
   Do You Need Microservices?
      Scaling Challenges
         Are we struggling to get changes made?
         Does our developer experience suck?
      Technical Reasons
         Do we have different compliance or security requirements for some parts of our system?
         Do we need to scale to support load?
         Are we facing operational challenges?
         Would we benefit from making a different technology choice for some part of our system?
   Spotting Potential Pitfalls
      Organizational Structure and Culture
      Software Delivery Approach
B. Recommended Reading
   Part I: Context
   Part II: Organizational Structure and Culture
   Part III: Building and Operating
Index