Databricks Data Intelligence Platform : Unlocking the GenAI Revolution

About the Authors
About the Technical Reviewers
Chapter 1: Databricks Platform: From Lakehouse to Data Intelligence Platform
	Data Platforms: Historical Perspective
	Emergence of the Lakehouse
	What Is a Lakehouse?
	What Is the Databricks Lakehouse?
	Key Features of the Databricks Lakehouse Platform
	Introducing the Databricks Data Intelligence Platform
	Conclusion
Chapter 2: Databricks Platform Overview
	Key Terminology
	Databricks Compute or Clusters
		Interactive or All-Purpose Clusters
		Job Cluster
		SQL Warehouse
	Databricks All-Purpose Cluster Setup
		Policy
		Access Mode
		Databricks Runtime Version
		Autoscaling and Autotermination
		Tags
		Spot Instances
		Cluster Pools
	Cluster Sizing Considerations and Best Practices
	Databricks Notebooks
		Debugging
		Serverless in Notebook
		Databricks Widgets
	Library Management
	External Databricks Connectivity
		Databricks CLI
		Databricks REST API
		Databricks Terraform
	Conclusion
Chapter 3: Data Ingestion in Lakehouse
	Introduction
	Cloud Ingestion
	Delta Ingestion
		Auto Loader
		COPY INTO
	Conclusion
Chapter 4: Delta Lake - Deep Dive
	The Challenges of Other Formats
	What Is Delta Lake?
	Delta Lake: Medallion Architecture
	Delta Lake Key Features
		Update, Delete, and Upserts in Delta Table
		Schema Evolution
	Time Travel
	Clone Delta Tables
	Generated Column
	Change Data Feed
	Universal Format
	Delta Optimization
	Liquid Clustering
	Working with Liquid Clustering
	Current Limitations
	Predictive I/O
		ML/AI to the Rescue
	Conclusion
Chapter 5: Data Governance with Unity Catalog
	What Is Databricks Unity Catalog?
	Unity Catalog: Before and After
	Unity Catalog Hierarchy
	Unity Catalog Admin Roles
		Getting Started with Unity Catalog
		Create a Metastore
	Organizing Data in Unity Catalog
	Key Features of Unity Catalog
		Centralized Metadata and User Management
		Centralized Access Controls
	Data Lineage
	Data Access Auditing
	Data Search and Discovery
	Row-Level Security and Column-Level Masking
		Row Filters
		Create a Row Filter
		Apply the Row Filter to a Table
		Column Masks
		Dynamic Views vs. Row Filters and Column Masks
	Delta Sharing
		An Open Standard for Data Sharing
		How Delta Sharing Works
	Conclusion
Chapter 6: Data Engineering Part 1: Orchestrating Data Pipelines Using Databricks Workflows
	Databricks Workflow Jobs
	Databricks Jobs and Tasks
		Configure Databricks Job Tasks: Task-Level Parameters
		Configure Databricks Job Tasks: Job-Level Parameters
	Advanced Workflow Features
	Monitoring Data Pipelines
	Conclusion
Chapter 7: Data Engineering Part 2: Delta Live Tables
	What Is Delta Live Tables?
		Data Ingestion Using DLT
		Change Data Capture with DLT
		Delta Live Tables Expectations
	Creating a DLT Pipeline
	Logging and Monitoring
	Enhanced Autoscaling
	Runtime Channels
	Example: A Retail Sales Pipeline
		Streaming Pipeline
		Data Validation
		Data Lineage
		Validation Dashboard
	Conclusion
Chapter 8: Data Warehousing with DBSQL
	What Is Databricks SQL?
	SQL Warehouses
		Photon
		SQL Editor
		Introduction to AI/BI Dashboards
		Alerts
		Query History and Profile
		Serverless Compute
	Constraints in DBSQL
		Constraints on Databricks
		Enforced Constraints
		Informational Constraints: Primary Key Foreign Key
	Streaming Tables and Materialized Views
		Streaming Tables
	Materialized Views
		Create a Materialized View
		Refresh a Materialized View
		Lakehouse Federation
		AI Functions in DBSQL
		Consume LLM Models in DBSQL
		Custom Functions Backed by a Serverless Serving Endpoint
		Integrate BI Tools with Databricks
		Publish to PowerBI Online from Databricks
	Connect Power BI Desktop to Databricks
	Conclusion
Chapter 9: Machine Learning Operations Using Databricks
	Machine Learning with Databricks
		Experiments
		What Is the Glass Box Approach to Automated Machine Learning?
	Machine Learning Lifecycle: MLOps
		ML Example: Predicting Flight Delays with Databrick’s AutoML
			Prepare Data
			Exploratory Data Analysis
			Feature Engineering
		Data Exploration at Scale
			Pandas Profiling
			Data Summarization Using dbutils
		Feature Store
			Why Use Databricks Feature Store?
		Model Building
			Model Training
			Validation
		Deploy Model
			Deployment
				Model Serving/Inferencing
			Monitoring
				Lakehouse Monitoring
				Why Profiling?
				Deep Dive into Lakehouse Monitoring Output Tables
		MLOps Best Practices
		Conclusion
Chapter 10: Generative AI with Databricks
	What Is Generative AI?
	Databricks Generative AI
	The GenAI Journey
	Prompt Engineering
		Mosaic AI Playground
		Use Cases
			Sentiment Analysis
			Unstructured Text Parsing
			Summarization
			Document Q&A
	Retrieval Augmented Generation
		Similarity Search: The Magic Behind the Scenes
		A Practical Example for RAG: Using Structured Data
		Step 1: Feature and Function Serving
		Step 2: Calculate Embedding and Sync to a Vector Database
		Step 3: Create a LangChainTool to Perform Various Tasks
		Step 4: MLflow LLM Evaluation
	Mosaic AI Fine-Tuning API
		Fine-Tuning Example
	Pre-Training
		A Case Study of AI2’s OLMo, a Truly Open-Source Large Language Model
	Gen AI Pricing
		What Are Tokens and Tokenizers?
	Conclusion
Chapter 11: Large Language Model Operations
	Machine Learning Operations
	Large Language Model Operations
	Components of LLMOps
	Deep Dive into Each Process
		Prompt Engineering
			Prompt Templates
			Chain of Thoughts
		Retrieval Augmented Generation
		Model Fine-Tuning
		Model Pretraining
	A Case Study of AI2’s OLMo, a Truly Open-Source Large Language Model
		Model Governance
			MLflow Deployments Server
		LLM as a Judge
		Model Packaging and Deployment
			LangChain Flavor with MLflow
	Conclusion
Chapter 12: Mosaic AI Agent Framework: Creating Quality AI Agents
	Part 0: The Installations
	Part 1: LangChain Parametrization
	Part 2: MLflow Evaluation
	Part 3: Model Development
	Part 4: Deployment
	Evaluation Example
	Conclusion
		Beyond LangChain
Chapter 13: DBRX: Creating an LLM from Scratch Using Databricks
	What Is DBRX?
	The DBRX Benchmarks
	DBRX Architecture
		Shortcomings of the Transformer Architecture
		Mixture of Experts
		MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
		Fine-Grained MoE
	The MosaicML Stack
	Distributed GPU Training
	Model Serving
	Using DBRX on Databricks
	Conclusion
Chapter 14: The Databricks Data Intelligence Platform
	Databricks IQ
	Deep Dive into Databricks IQ
		Databricks Assistant
			Generate Code in Any Language
			Autocomplete Code or Queries
			Code Conversion
			Code Explanation
			Code Fixing
		AI-Powered Governance
		Search and Discovery
			Intelligent Search
		AI/BI Genie (Previous Data Rooms)
		How to Set Up Genie
		Conclusion
Chapter 15: Databricks CI/CD
	What Is CI/CD?
	Stages of CI/CD
	Introduction to Databricks Repos
	Databricks UI vs. Git Terminologies
	Databricks Asset Bundles
	Case Study: Databricks MLOps Stack
	Conclusion
Chapter 16: Databricks Pricing and Observability Using System Tables
	Costs Associated with the Databricks Platform
	Cloud Infrastructure Costs
	Databricks Pricing
		What Are Databricks Units?
		SQL Warehouse Pricing
	Databricks Cost Management Best Practices
	Databricks Observability: System Tables
		Introduction to System Tables
		Common Schemas/Tables Available with System Tables
		System Table: Billing Usage Example
	Conclusion
Chapter 17: Databricks Platform Security and Compliance
	Databricks Architecture
	Azure Databricks Deployment
		Capacity Planning
		VNET Injection or Bring Your Own VNET
		Secure Cluster Connectivity (No Public IP/NPIP)
		Azure Private Link for Back-End and Front-End Connections
		Encryption and Auditing
		Customer Managed Keys
	Identity and Access
		SSO and Multifactor Authentication
		IP Access Lists
		Role-Based Access Control
		Token Management API
	Security Analysis Tool
	Databricks Security Best Practices
	Conclusion
Chapter 18: Spark Structured Streaming: A Comprehensive Guide
	Spark Streaming
	Structured Streaming
	What Is Continuous Processing?
	Triggers
	Output Modes
	Windowed Grouped Aggregation
	State Management
	Late-Arrival Handling: Watermark
	Auto Loader
	Project Lightspeed
		Advanced State Management
		Use Case: E-commerce Operation
	Structured Streaming Best Practices
	Conclusion
Chapter 19: From Ideation to Creation: A Walk- Through of Building a GenAI Application
	The Problem Statement
	Data Generation: Source
	Data Ingestion: Ingest
	Data Transformation: Transform
		Using Serverless SQL for Transformation
	Machine Learning Model for Diabetes Complication Classification: Query and Process
	Generative AI: Serve
		Where Do We Start?
	Monitoring Dashboard: Analysis
	Conclusion
Index