DevOps for Databases: A practical guide to applying DevOps best practices to data-persistent technologies

DevOps for Databases
Contributors
About the author
About the reviewer
Preface
   Who this book is for
   What this book covers
   To get the most out of this book
   Conventions used
   Get in touch
   Share Your Thoughts
   Download a free PDF copy of this book
Part 1: Database DevOps
1
Data at Scale with DevOps
   The modern data landscape
      How do we generate data today?
   Why speed matters
   Data management strategies
   The early days of DevOps
   SRE versus DevOps
   Engineering principles
   Objectives – SLOs/SLIs
   Summary
2
Large-Scale Data-Persistent Systems
   A brief history of data
      The early days of computing
      The rise of relational databases
      Conclusion
   Database evolution
      Hierarchical database models
      Network database model
      Relational databases
      OO databases
      NoSQL database paradigms
   Data warehouses
      Architecture
      Data modeling
      Integration
   Data lakes
      Architecture
      Data ingestion and processing
      Storage and retrieval
      Security considerations
      Conclusion
   Summary
3
DBAs in the World of DevOps
   The continuously evolving role of the DBA
      The rise of data architecture and integration
   The emergence of cloud computing and big data
      The shift to DevOps
      Conclusion
   DevOps and DBAs
   The role of the database expert in a DevOps-conscious team
      Designing and implementing databases
      Ensuring high availability and disaster recovery
      Performance tuning
      Security and compliance
      Automation
   A proven methodology with quantifiable benefits
      Faster time to market
      Improved quality
      Reduced downtime
      Increased scalability
      Improved security
   Summary
Part 2: Persisting Data in the Cloud
4
Cloud Migration and Modern Data(base) Evolution
   What is cloud migration (and why are companies doing it)?
      The importance of cloud migration
      Steps to keep in mind before cloud migration
      Key milestones through cloud migration
   Types of cloud migrations
      Lift-and-shift migration
      Lift-and-reshape migration
      Refactoring migration
      Hybrid cloud migration
      Multi-cloud migration
   The process of cloud migration
      Monolithic or distributed database systems?
   What can a database expert help with during cloud migration?
   Data migration strategies and their types
      The Big Bang migration strategy
      The phased migration strategy
      The parallel migration strategy
      The hybrid migration strategy
      The reverse migration strategy
      The ETL strategy
      The replication strategy
   Why are data migration strategies important during a database cloud migration project?
      Taking your migration journey one step further
   Summary
5
RDBMS with DevOps
   Embracing DevOps
      Provisioning and configuration management
      Monitoring and alerting
      Backup and disaster recovery
      Performance optimization
      DevSecOps
   Summary
6
Non-Relational DMSs with DevOps
   Activities and challenges
   Data modeling
      Denormalization
      Nested and dynamic data
      Data denormalization
   Schema management
      Schemaless data modeling
      Dynamic schema evolution
      Consistency and concurrency control
   Deployment automation
      Deployment of multiple database engines
      Backup and disaster recovery
      Capacity planning and scaling
   Performance tuning
      Data modeling for performance
      Distributed query optimization
      Network latency and data transfer
   Data consistency
      Lack of transactions
      Eventual consistency
      Data sharding
   Security
      Limited access control
      Distributed denial of service attacks
      Lack of encryption
   Anti-patterns (what not to do…)
      Overusing or misusing denormalization
      Ignoring or underestimating data consistency
      Failing to secure a database
      Overlooking performance tuning
      Neglecting to plan for growth
   Summary
7
AI, ML, and Big Data
   Definitions and applications of AI, ML, and big data
      The relationship between AI, ML, and big data
      The role of DevOps and engineering in AI, ML, and big data
      Challenges of AI, ML, and big data
   A deep dive into big data as a DevOps data expert
      Big data infrastructure
      Challenges with big data
   A deep dive into ML as a DevOps data expert
      How ML works
      How ML differs from traditional software applications
      Challenges with ML for a DevOps data expert
   A deep dive into AI as a DevOps data- expert
      Amazon SageMaker
      Google Cloud AI platform
      Microsoft Azure Machine Learning
      Challenges with AI for a DevOps data expert
   Summary
Part 3: The Right Tool for the Job
8
Zero-Touch Operations
   Traditional versus zero-touch approaches
      Automated configuration management
      Automated release management
      Automated monitoring and alerting
   Increased operational efficiency
      Automated database provisioning
      Automated backup and recovery
   Improved reliability and consistency
      Automated configuration management
      Automated release management
   Accelerated deployment and time-to-market
      CI/CD pipelines
      IaC and orchestration
   Enhanced scalability and elasticity
      Automated resource provisioning
      Container orchestration
   Reduced downtime and faster recovery
      Automated monitoring and alerting
      Streamlined recovery processes
   Improved compliance and security
      Automated security configurations
      Automated compliance checks
   Sanity-checking our approach
      Conclusion on ROI
   Summary
9
Design and Implementation
   Designing data-persistence technologies
      Database design principles
      RDBMS versus NoSQL versus NewSQL
   Implementing data-persistence technologies
      Installation, configuration, and management of database systems
      Practical example – PostgreSQL database server installation, configuration, and management
      Disaster recovery planning
      Practical example – MongoDB replication and automatic failover
   Database provisioning and Infrastructure as Code
      Practical example – using Terraform to script the setup of a SQL Server database
   Database version control and CI/CD
      Importance of database version control
      Practical example – using Liquibase to manage database schema changes
      Role of the DevOps DBA in CI/CD pipelines
      Practical example – Jenkins pipeline with database migrations using Flyway
   Database performance tuning
      Importance of performance tuning and common strategies
      Practical example – optimizing a poorly performing query in Oracle
   Security and compliance
      Importance of security in database management
      Common security measures
      Practical example – best practices for securing a MySQL database and ensuring GDPR compliance
   Collaboration and communication
   Summary
10
Database Automation
   Autonomous database management
      Self-driving databases – a new horizon in DBMs
      Automation of database administration tasks
      The implications of self-driving databases
      Challenges and future directions
      Conclusion
   The revolution of performance tuning – from manual to autonomous
      Understanding performance tuning
      The need for automated performance tuning
      Technological foundations of automated performance tuning
      The mechanics of automated performance tuning
      The implications of automated performance tuning
      Challenges and future directions
      Conclusion
   Automated data lineage tracking – a new era of transparency in data management
      Understanding data lineage
      The evolution from manual to automated data lineage tracking
      The technological foundations of automated data lineage tracking
      The process of automated data lineage tracking
      The implications of automated data lineage tracking
      Challenges and future directions
      Conclusion
   Data privacy automation – advancing the frontier of privacy compliance in the digital age
      Understanding data privacy
      The challenges of data privacy
      Data masking and anonymization
      The advent of data privacy automation
      Technological underpinnings of data privacy automation
      The process of data privacy automation
      The benefits and implications of data privacy automation
      Conclusion
   Automated data discovery and cataloging – unveiling the hidden treasures in today’s data landscape
      Understanding data discovery and cataloging
      The growing need for automation in data discovery and cataloging
      What is automated data discovery and cataloging?
      The key features of automated data discovery and cataloging tools
      The process of automated data discovery and cataloging
      The benefits and implications of automated data discovery and cataloging
      Conclusion
   The ascendancy of DBaaS – transforming business efficiency and data utilization in the digital age
      Understanding DBaaS
      The mechanics of DBaaS
      The impact of DBaaS on business operations
      DBaaS – the future of database management
      Conclusion
   The emergence of serverless databases – revolutionizing DBaaS through on-demand scalability and cost efficiency
      Understanding serverless databases
      Why serverless databases? The driving forces
      How serverless databases work
      The impact of serverless databases on business operations
      The future of serverless databases
      Conclusion
   Summary
Part 4: Build and Operate
11
End-to-End Ownership Model – a Theoretical Case Study
   End-to-end ownership – a case study
   Adoption of the end-to-end ownership model
   Setting the stage
      Project initiation
      Formation of cross-functional teams
      Defining end-to-end ownership
   Design and development phase
      Collaborative design and planning
      Agile development practices
      CI and continuous testing
   Deployment and release
      IaC
      CD pipelines
      Canary releases and feature flags
   Monitoring and IM
      Proactive monitoring and alerting
      IR and post-mortems
      Continuous improvement
   Feedback and iteration
      Gathering user feedback
      Prioritizing and implementing changes
      A/B testing and experiments
   Scaling and challenges
      Scaling the end-to-end ownership model
      Managing dependencies
      Balancing autonomy and alignment
   Summary
12
Immutable and Idempotent Logic – A Theoretical Case Study
   Introduction to immutable and idempotent logic
   Immutable logic in data-persisting technologies
      Understanding immutability in the context of data storage
      Benefits and use cases of immutable data storage
      Examples of immutable data storage approaches
      Implementing immutable logic with data-persisting technologies
   Idempotent logic in data-persisting technologies
      Introduction to idempotent operations and their significance
      Examples of idempotent operations in data persistence
      Ensuring idempotency in data processing and transformations
   Practical examples and use cases
      Immutable and idempotent logic in relational databases
      Immutable and idempotent approaches in NoSQL databases
      Immutable and idempotent patterns in distributed storage systems
   Considerations and best practices
      Performance and scalability implications of immutable and idempotent approaches
      Data consistency and integrity considerations
      Handling failures and retries with idempotent logic
      Managing data evolution and schema changes with immutability
   Future trends and challenges
      Emerging technologies and advancements in data persistence
      Integrating immutable and idempotent logic with cloud-native architectures
      Addressing complexities and trade-offs in large-scale data persistence systems
   Summary
13
Operators and Self-Healing Data Persistent Systems
   Self-healing systems
      Components of a self-healing system
      Importance of self-healing systems
      Risks and limitations
      Technical example of each core principle of a self-healing system
   Operators in Kubernetes
      Overview of Kubernetes and containerization
      Understanding operators
      Operator frameworks and ecosystem
      Benefits of operators in Kubernetes
      Use cases of operators in Kubernetes
   Self-healing databases
      Traditional database challenges
      Self-healing mechanisms in databases
      Benefits of self-healing databases
      Risks and limitations
   Factors influencing self-healing in different databases
      Relational databases
      NoSQL databases
      NewSQL databases
      Time-series databases
   Self-healing in Kubernetes – implementation and best practices
      Key components for self-healing in Kubernetes
      Implementing self-healing in Kubernetes – best practices
      Challenges and considerations
      Conclusion
   Case studies – self-healing databases in Kubernetes
      Case study 1 – MySQL Operator
      Case study 2 – MongoDB Operator
      Case study 3 – Cassandra Operator
   Benefits of self-healing databases in Kubernetes
   Challenges and future directions
      Challenges in self-healing systems
      Future directions
   Summary
14
Bringing Them Together
   Alex’s AI journey
      Introduction and project assignment
      Software and infrastructure architecture decisions
      Relational versus non-relational databases
      Implementing caching, data lakes, and data warehouses
      Security concerns and solutions
      First status update
   Implementation
      Utilizing DevOps and SRE methodologies
      The power of immutable and idempotent logic
      Embracing zero-touch automation
      Update 2
      Implementing self-healing systems
      Implementing load balancers and scaling
      Update 3
   Observability and operations
      The art of canary deployments
      Database scaling
      Security and compliance in operations
      Update 4
      Version-controlled environment variables
      Update 5
   Lessons learned and future directions
   Summary
Part 5: The Future of Data
15
Specializing in Data
   Mastering data – bridging the gap between IT and business
      Data-driven decision-making for business executives
      Building a data-driven culture – an enterprise perspective
   My first experience, Unix – 2009
   The first signs of DevOps – 2010s
      Support and software engineering in 2012
   My first SRE team – 2015
   Steep learning curves – 2017
   Putting it all into practice – 2019
   The landscape of 2023 – data and DevOps united
      Integration of DevOps with data engineering
      DataOps – revolutionizing data analytics pipelines
      MLOps – bridging the gap between ML development and operations
      AI-powered DevOps/SRE
      Application of SRE principles to data systems
      DevSecOps – security in the age of data
   Summary
16
The Exciting New World of Data
   Part 1 – the future of data-persisting technologies
      The evolution of current data-persisting technologies
      Emerging data-persisting technologies
      Future use cases and challenges
   Part 2 – anticipated changes in AI/ML DevOps
      Advancements in MLOps
      Future use cases and challenges
      Career impacts and future skill requirements
   Part 3 – evolving trends in SRE
      Changes in the SRE landscape
      The role of SRE in future IT operations
   Part 4 – career outlook and emerging skill sets in SRE
      SRE careers – opportunities, skills, and preparation
      Career opportunities in SRE
      Skill set requirements
      Preparing for a career in SRE
      Innovations in data persistence technologies
      The current state of data persistence technologies
      Future outlook – next-generation data persistence technologies
      Preparing for the future – skills and strategies in data persistence technologies
   Part 5 – the future of designing, building, and operating cutting-edge systems
      Emerging technologies in system design and development
      Potential implications and challenges in designing, building, and operating cutting-edge systems
      Strategies for success – preparing for the future of designing, building, and operating cutting-edge systems
   Summary
Index
   Why subscribe?
Other Books You May Enjoy
   Packt is searching for authors like you
   Share Your Thoughts
   Download a free PDF copy of this book