The Self-Service Data Roadmap

Cover
Copyright
Table of Contents
Preface
	Conventions Used in This Book
	Using Code Examples
	O’Reilly Online Learning
	How to Contact Us
Chapter 1. Introduction
	Journey Map from Raw Data to Insights
		Discover
		Prep
		Build
		Operationalize
	Defining Your Time-to-Insight Scorecard
	Build Your Self-Service Data Roadmap
Part I. Self-Service Data Discovery
	Chapter 2. Metadata Catalog Service
		Journey Map
			Understanding Datasets
			Analyzing Datasets
			Knowledge Scaling
		Minimizing Time to Interpret
			Extracting Technical Metadata
			Extracting Operational Metadata
			Gathering Team Knowledge
		Defining Requirements
			Technical Metadata Extractor Requirements
			Operational Metadata Requirements
			Team Knowledge Aggregator Requirements
		Implementation Patterns
			Source-Specific Connectors Pattern
			Lineage Correlation Pattern
			Team Knowledge Pattern
		Summary
	Chapter 3. Search Service
		Journey Map
			Determining Feasibility of the Business Problem
			Selecting Relevant Datasets for Data Prep
			Reusing Existing Artifacts for Prototyping
		Minimizing Time to Find
			Indexing Datasets and Artifacts
			Ranking Results
			Access Control
		Defining Requirements
			Indexer Requirements
			Ranking Requirements
			Access Control Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Push-Pull Indexer Pattern
			Hybrid Search Ranking Pattern
			Catalog Access Control Pattern
		Summary
	Chapter 4. Feature Store Service
		Journey Map
			Finding Available Features
			Training Set Generation
			Feature Pipeline for Online Inference
		Minimize Time to Featurize
			Feature Computation
			Feature Serving
		Defining Requirements
			Feature Computation
			Feature Serving
			Nonfunctional Requirements
		Implementation Patterns
			Hybrid Feature Computation Pattern
			Feature Registry Pattern
		Summary
	Chapter 5. Data Movement Service
		Journey Map
			Aggregating Data Across Sources
			Moving Raw Data to Specialized Query Engines
			Moving Processed Data to Serving Stores
			Exploratory Analysis Across Sources
		Minimizing Time to Data Availability
			Data Ingestion Configuration and Change Management
			Compliance
			Data Quality Verification
		Defining Requirements
			Ingestion Requirements
			Transformation Requirements
			Compliance Requirements
			Verification Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Batch Ingestion Pattern
			Change Data Capture Ingestion Pattern
			Event Aggregation Pattern
		Summary
	Chapter 6. Clickstream Tracking Service
		Journey Map
		Minimizing Time to Click Metrics
			Managing Instrumentation
			Event Enrichment
			Building Insights
		Defining Requirements
			Instrumentation Requirements Checklist
			Enrichment Requirements Checklist
		Implementation Patterns
			Instrumentation Pattern
			Rule-Based Enrichment Patterns
			Consumption Patterns
		Summary
Part II. Self-Service Data Prep
	Chapter 7. Data Lake Management Service
		Journey Map
			Primitive Life Cycle Management
			Managing Data Updates
			Managing Batching and Streaming Data Flows
		Minimizing Time to Data Lake Management
			Requirements
		Implementation Patterns
			Data Life Cycle Primitives Pattern
			Transactional Pattern
			Advanced Data Management Pattern
		Summary
	Chapter 8. Data Wrangling Service
		Journey Map
		Minimizing Time to Wrangle
			Defining Requirements
			Curating Data
			Operational Monitoring
		Defining Requirements
		Implementation Patterns
			Exploratory Data Analysis Patterns
			Analytical Transformation Patterns
		Summary
	Chapter 9. Data Rights Governance Service
		Journey Map
			Executing Data Rights Requests
			Discovery of Datasets
			Model Retraining
		Minimizing Time to Comply
			Tracking the Customer Data Life Cycle
			Executing Customer Data Rights Requests
			Limiting Data Access
		Defining Requirements
			Current Pain Point Questionnaire
			Interop Checklist
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Sensitive Data Discovery and Classification Pattern
			Data Lake Deletion Pattern
			Use Case–Dependent Access Control
		Summary
Part III. Self-Service Build
	Chapter 10. Data Virtualization Service
		Journey Map
			Exploring Data Sources
			Picking a Processing Cluster
		Minimizing Time to Query
			Picking the Execution Environment
			Formulating Polyglot Queries
			Joining Data Across Silos
		Defining Requirements
			Current Pain Point Analysis
			Operational Requirements
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Automatic Query Routing Pattern
			Unified Query Pattern
			Federated Query Pattern
		Summary
	Chapter 11. Data Transformation Service
		Journey Map
			Production Dashboard and ML Pipelines
			Data-Driven Storytelling
		Minimizing Time to Transform
			Transformation Implementation
			Transformation Execution
			Transformation Operations
		Defining Requirements
			Current State Questionnaire
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Implementation Pattern
			Execution Patterns
		Summary
	Chapter 12. Model Training Service
		Journey Map
			Model Prototyping
			Continuous Training
			Model Debugging
		Minimizing Time to Train
			Training Orchestration
			Tuning
			Continuous Training
		Defining Requirements
			Training Orchestration
			Tuning
			Continuous Training
			Nonfunctional Requirements
		Implementation Patterns
			Distributed Training Orchestrator Pattern
			Automated Tuning Pattern
			Data-Aware Continuous Training
		Summary
	Chapter 13. Continuous Integration Service
		Journey Map
			Collaborating on an ML Pipeline
			Integrating ETL Changes
			Validating Schema Changes
		Minimizing Time to Integrate
			Experiment Tracking
			Reproducible Deployment
			Testing Validation
		Defining Requirements
			Experiment Tracking Module
			Pipeline Packaging Module
			Testing Automation Module
		Implementation Patterns
			Programmable Tracking Pattern
			Reproducible Project Pattern
		Summary
	Chapter 14. A/B Testing Service
		Journey Map
		Minimizing Time to A/B Test
			Experiment Design
			Execution at Scale
			Experiment Optimization
		Implementation Patterns
			Experiment Specification Pattern
			Metrics Definition Pattern
			Automated Experiment Optimization
		Summary
Part IV. Self-Service Operationalize
	Chapter 15. Query Optimization Service
		Journey Map
			Avoiding Cluster Clogs
			Resolving Runtime Query Issues
			Speeding Up Applications
		Minimizing Time to Optimize
			Aggregating Statistics
			Analyzing Statistics
			Optimizing Jobs
		Defining Requirements
			Current Pain Points Questionnaire
			Interop Requirements
			Functionality Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Avoidance Pattern
			Operational Insights Pattern
			Automated Tuning Pattern
		Summary
	Chapter 16. Pipeline Orchestration Service
		Journey Map
			Invoke Exploratory Pipelines
			Run SLA-Bound Pipelines
		Minimizing Time to Orchestrate
			Defining Job Dependencies
			Distributed Execution
			Production Monitoring
		Defining Requirements
			Current Pain Points Questionnaire
			Operational Requirements
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Dependency Authoring Patterns
			Orchestration Observability Patterns
			Distributed Execution Pattern
		Summary
	Chapter 17. Model Deploy Service
		Journey Map
			Model Deployment in Production
			Model Maintenance and Upgrade
		Minimizing Time to Deploy
			Deployment Orchestration
			Performance Scaling
			Drift Monitoring
		Defining Requirements
			Orchestration
			Model Scaling and Performance
			Drift Verification
			Nonfunctional Requirements
		Implementation Patterns
			Universal Deployment Pattern
			Autoscaling Deployment Pattern
			Model Drift Tracking Pattern
		Summary
	Chapter 18. Quality Observability Service
		Journey Map
			Daily Data Quality Monitoring Reports
			Debugging Quality Issues
			Handling Low-Quality Data Records
		Minimizing Time to Insight Quality
			Verify the Accuracy of the Data
			Detect Quality Anomalies
			Prevent Data Quality Issues
		Defining Requirements
			Detection and Handling Data Quality Issues
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Accuracy Models Pattern
			Profiling-Based Anomaly Detection Pattern
			Avoidance Pattern
		Summary
	Chapter 19. Cost Management Service
		Journey Map
			Monitoring Cost Usage
			Continuous Cost Optimization
		Minimizing Time to Optimize Cost
			Expenditure Observability
			Matching Supply and Demand
			Continuous Cost Optimization
		Defining Requirements
			Pain Points Questionnaire
			Functional Requirements
			Nonfunctional Requirements
		Implementation Patterns
			Continuous Cost Monitoring Pattern
			Automated Scaling Pattern
			Cost Advisor Pattern
		Summary
Index
About the Author
Colophon