Fundamentals of Data Observability: Implement Trustworthy End-to-End Data Solutions

Copyright
Table of Contents
Preface
	Overview of the Book
	Who Should Read This Book
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
	Acknowledgments
Part I. Introducing Data Observability
	Chapter 1. Introducing Data Observability
		Scaling Data Teams
			Challenges of Scaling Data Teams
			Segregated Roles and Responsibilities and Organizational Complexity
			Anatomy of Data Issues and Consequences
			Impact of Data Issues on Data Team Dynamics
			Scaling AI Roadblocks
		Challenges with Current Data Management Practices
			Effects of Data Governance at Scale
			Data Observability to the Rescue
			The Areas of Observability
		How Data Teams Can Leverage Data Observability Now
			Low Latency Data Issues Detection
			Efficient Data Issues Troubleshooting
			Preventing Data Issues
			Decentralized Data Quality Management
			Complementing Existing Data Governance Capabilities
			The Future and Beyond
		Conclusion
	Chapter 2. Components of Data Observability
		Channels of Data Observability Information
			Logs
			Traces
			Metrics
		Observations Model
			Physical Space
			Server
			User
			Static Space
			Dynamic Space
		Expectations
			Rules
			Automatic Anomaly Detection
			Prevent Garbage In, Garbage Out
		Conclusion
	Chapter 3. Roles of Data Observability in a Data Organization
		Data Architecture
			Where Does Data Observability Fit in a Data Architecture?
			Data Architecture with Data Observability
		How Data Observability Helps with Data Engineering Undercurrents
			Security
			Data Management
		Support for Data Mesh’s Data as Products
		Conclusion
Part II. Implementing Data Observability
	Chapter 4. Generate Data Observations
		At the Source
		Generating Data Observations at the Source
		Low-Level API in Python
			Description of the Data Pipeline
			Definition of the Status of the Data Pipeline
			Data Observations for the Data Pipeline
			Generate Contextual Data Observations
			Generate Data-Related Observations
			Generate Lineage-Related Data Observations
			Wrap-Up: The Data-Observable Data Pipeline
			Using Data Observations to Address Failures of the Data Pipeline
		Conclusion
	Chapter 5. Automate the Generation of Data Observations
		Abstraction Strategies
			Event Listeners
			Aspect-Oriented Programming
		High-Level Applications
			No-Code Applications
			Low-Code Applications
		Differences Among Monitoring Alternatives
		Conclusion
	Chapter 6. Implementing Expectations
		Introducing Expectations
			Shift-Left Data Quality
			Corner Cases Discovery
			Lifting Service Level Indicators
			Using Data Profilers
		Maintaining Expectations
		Overarching Practices
			Fail Fast and Fail Safe
			Simplify Tests and Extend CI/CD
		Conclusion
Part III. Data Observability in Action
	Chapter 7. Integrating Data Observability in Your Data Stack
		Ingestion Stage
			Ingestion Stage Data Observability Recipes
			Airbyte Agent
		Transformation
			Transformation Stage Data Observability Recipes
			Apache Spark
			dbt Agent
		Serving
			Recipes
			BigQuery in Python
			Orchestrated SQL with Airflow
		Analytics
			Machine Learning Recipes
			Business Intelligence Recipes
		Conclusion
	Chapter 8. Making Opaque Systems Translucent
		Data Translucence
		Opaque Systems
			SaaS
			Don’t Touch It; It (Kinda) Works
			Inherited Systems
		Strategies for Data Translucence
			Strategies
			The Data Observability Connector
			Example: Building a dbt Data Observability Connector (SaaS)
		Conclusion
Afterword: Future Observations
	Unification of Processing
	Generative Milestones
	Trustable Expanded Creativity
	Conclusion
Index
About the Author
Colophon