Optimizing Databricks Workloads: Harness the power of Apache Spark in Azure and maximize the performance of modern big data workloads

Cover\nTitle Page\nCopyright and credit page\nContributors\nAbout the reviewer\nTable of Contents\nPreface\nSection 1: Introduction to Azure Databricks\nChapter 1: Discovering Databricks\n	Technical requirements\n	Introducing Spark fundamentals\n	Introducing Databricks\n		Creating an Azure Databricks workspace\n		Core Databricks concepts\n		Creating a Spark cluster\n		Databricks notebooks\n		Databricks File System (DBFS)\n		Databricks jobs\n		Databricks Community\n	Learning about Delta Lake\n		Big data file formats\n		Understanding the transactional log\n		Delta Lake in action\n	Summary\nChapter 2: Batch and Real-Time Processing in Databricks\n	Technical requirements\n	Differentiating batch versus real-time processing\n	Mounting Azure Data Lake in Databricks\n		Creating an Azure Data Lake instance\n		Accessing Azure Data Lake in Databricks\n	Working with batch processing\n		Reading data\n		Checking row count\n		Selecting columns\n		Filtering data\n		Dropping columns\n		Adding or replacing columns\n		Printing schema\n		Renaming a column\n		Dropping duplicate rows\n		Limiting output rows\n		Sorting rows\n		Grouping data\n		Visualizing data\n		Writing data to a sink\n	Batch ETL process demo\n	Learning Structured Streaming in Azure Databricks\n		Structured Streaming concepts\n		Managing streams\n		Sorting data\n		Productionizing Structured Streaming\n	Summary\nChapter 3: Learning about Machine Learning and Graph Processing in Databricks\n	Technical requirements\n	Learning about ML components in Databricks\n	Practicing ML in Databricks\n		Environment setup\n		EDA\n		ML\n	Learning about MLflow\n	Learning about graph analysis in Databricks\n	Summary\nSection 2: Optimization Techniques\nChapter 4: Managing Spark Clusters\n	Technical requirements\n	Designing Spark clusters\n		Understanding cluster types\n		Learning about spot instances\n		Learning about autoscaling in Spark clusters\n		Introducing Databricks Pools\n		Learning about Databricks runtime versions (DBRs)\n		Learning about automatic termination\n		Learning about cluster sizing\n	Learning about Databricks managed resource groups\n	Learning about Databricks Pools\n		Creating a pool\n		Attaching a cluster to the Pool\n		Following the best practices for Azure Databricks Pools\n	Using spot instances\n	Following the Spark UI\n		Understanding the Jobs section\n		Understanding the Stages section\n		Understanding the Storage section\n		Understanding the Environment section\n		Understanding the Executors section\n		Understanding the SQL section\n		Understanding the JDBC/ODBC Server section\n		Understanding the Structured Streaming section\n	Summary\nChapter 5: Big Data Analytics\n	Technical requirements\n	Understanding the collect() method\n	Understanding the use of inferSchema\n		Experiment 1\n		Experiment 2\n	Learning to differentiate CSV and Parquet\n	Learning to differentiate Pandas and Koalas\n	Understanding built-in Spark functions\n	Learning column predicate pushdown\n	Learning partitioning strategies in Spark\n		Understanding Spark partitions\n		Understanding Hive partitions\n	Understanding Spark SQL optimizations\n	Understanding bucketing in Spark\n	Summary\nChapter 6: Databricks Delta Lake\n	Technical requirements\n	Working with the OPTIMIZE and ZORDER commands\n	Using Auto Optimize\n		Understanding optimized writes\n		Understanding Auto Compaction\n	Learning about delta caching\n	Learning about dynamic partition pruning\n	Understanding bloom filter indexing\n	Summary\nChapter 7: Spark Core\n	Technical requirements\n	Learning about broadcast joins\n	Learning about Apache Arrow in Pandas\n	Understanding shuffle partitions\n	Understanding caching in Spark\n	Learning about AQE\n		Dynamically coalescing shuffle partitions\n		Dynamically switching join strategies\n		Dynamically optimizing skew joins\n	Summary\nSection 3: Real-World Scenarios\nChapter 8: Case Studies\n	Learning case studies from the manufacturing industry\n		Case study 1 – leading automobile manufacturing company\n		Case study 2 – international automobile manufacturing giant\n		Case study 3 – graph search in a chemical corporate firm\n		Case study 4 – real-time loyalty engine for a leading medical equipment manufacturer\n	Learning case studies from the media and entertainment industry\n		Case study 5 – HD Insights to Databricks migration for a media giant\n	Learning case studies from the retail and FMCG industry\n		Case study 6 – real-time analytics using IoT Hub for a retail giant\n	Learning case studies from the pharmaceutical industry\n		Case study 7 – pricing analytics for a pharmaceutical company\n	Learning case studies from the e-commerce industry\n		Case study 8 – migrating interactive analytical apps from Redshift to Postgres\n	Learning case studies from the logistics and supply chain industry\n		Case study 9 – accelerating intelligent insights with tailored big data analytics\n	Summary\nOther Books You May Enjoy\nIndex