Data Engineering with Google Cloud Platform: A practical guide to operationalizing scalable data analytics systems on GCP

Cover
Title page
Copyright and Credits
Contributors
Table of Contents
Preface
Section 1: Getting Started with Data Engineering with GCP
Chapter 1: Fundamentals of Data Engineering
	Understanding the data life cycle
		Understanding the need for a data warehouse
	Knowing the roles of a data engineer before starting
		Data engineer versus data scientist
		The focus of data engineers
	Foundational concepts for data engineering
		ETL concept in data engineering 
		The difference between ETL and ELT
		What is NOT big data?
		A quick look at how big data technologies store data
		A quick look at how to process multiple files using MapReduce
	Summary
	Exercise
	See also
Chapter 2: Big Data Capabilities on GCP
	Technical requirements
	Understanding what the cloud is
		The difference between the cloud and non-cloud era
		The on-demand nature of the cloud
	Getting started with Google Cloud Platform
		Introduction to the GCP console
		Practicing pinning services
		Creating your first GCP project
		Using GCP Cloud Shell
	A quick overview of GCP services for data engineering
		Understanding the GCP serverless service
		Service mapping and prioritization
		The concept of quotas on GCP services
		User account versus service account
	Summary
Section 2: Building Solutions with GCP Components
Chapter 3: Building a Data Warehouse in BigQuery
	Technical requirements
	Introduction to Google Cloud Storage and BigQuery
		BigQuery data location
	Introduction to the BigQuery console
		Creating a dataset in BigQuery using the console
		Loading a local CSV file into the BigQuery table
		Using public data in BigQuery
		Data types in BigQuery compared to other databases
		Timestamp data in BigQuery compared to other databases
	Preparing the prerequisites before developing our data warehouse
		Step 1: Access your Cloud shell
		Step 2: Check the current setup using the command line 
		Step 3: The gcloud init command
		Step 4: Download example data from Git
		Step 5: Upload data to GCS from Git
	Practicing developing a data warehouse
		Data warehouse in BigQuery – Requirements for scenario 1
		Steps and planning for handling scenario 1
		Data warehouse in BigQuery – Requirements for scenario 2
		Steps and planning for handling scenario 2
	Summary
	Exercise – Scenario 3
	See also
Chapter 4: Build Orchestration for Batch Data Loading Using Cloud Composer
	Technical requirements
	Introduction to Cloud Composer
	Understanding the working of Airflow
		Provisioning Cloud Composer in a GCP project
	Exercise: Build data pipeline orchestration using Cloud Composer
		Level 1 DAG – Creating dummy workflows
		Level 2 DAG – Scheduling a pipeline from Cloud SQL to GCS and BigQuery datasets
		Level 3 DAG – Parameterized variables
		Level 4 DAG – Guaranteeing task idempotency in Cloud Composer
		Level 5 DAG – Handling late data using a sensor
	Summary
Chapter 5: Building a Data Lake Using Dataproc
	Technical requirements
	Introduction to Dataproc
		A brief history of the data lake and Hadoop ecosystem
		A deeper look into Hadoop components
		How much Hadoop-related knowledge do you need on GCP?
		Introducing the Spark RDD and the DataFrame concept
		Introducing the data lake concept
		Hadoop and Dataproc positioning on GCP
	Exercise – Building a data lake on a Dataproc cluster
		Creating a Dataproc cluster on GCP
		Using Cloud Storage as an underlying Dataproc file system 
	Exercise: Creating and running jobs on a Dataproc cluster
		Preparing log data in GCS and HDFS
		Developing Spark ETL from HDFS to HDFS
		Developing Spark ETL from GCS to GCS
		Developing Spark ETL from GCS to BigQuery
	Understanding the concept of the ephemeral cluster
		Practicing using a workflow template on Dataproc
	Building an ephemeral cluster using Dataproc and Cloud Composer
	Summary 
Chapter 6: Processing Streaming Data with Pub/Sub and Dataflow
	Technical requirements
	Processing streaming data
		Streaming data for data engineers
		Introduction to Pub/Sub
		Introduction to Dataflow
	Exercise – Publishing event streams to cloud Pub/Sub  
		Creating a Pub/Sub topic
		Creating and running a Pub/Sub publisher using Python
		Creating a Pub/Sub subscription
	Exercise – Using Cloud Dataflow to stream data from Pub/Sub to GCS
		Creating a HelloWorld application using Apache Beam
		Creating a Dataflow streaming job without aggregation
		Creating a streaming job with aggregation
	Summary
Chapter 7: Visualizing Data for Making Data-Driven Decisions with Data Studio
	Technical requirements
	Unlocking the power of your data with Data Studio
	From data to metrics in minutes with an illustrative use case
		Understanding what BigQuery INFORMATION_SCHEMA is
		Exercise – Exploring the BigQuery INFORMATION_SCHEMA table using Data Studio
		Exercise – Creating a Data Studio report using data from a bike-sharing data warehouse
	Understanding how Data Studio can impact the cost of BigQuery
		What kind of table could be 1 TB in size?
		How can a table be accessed 10,000 times in a month?
	How to create materialized views and understanding how BI Engine works
		Understanding BI Engine
	Summary
Chapter 8: Building Machine Learning Solutions on Google Cloud Platform
	Technical requirements
	A quick look at machine learning
	Exercise – practicing ML code using Python
		Preparing the ML dataset by using a table from the BigQuery public dataset
		Training the ML model using Random Forest in Python
		Creating Batch Prediction using the training dataset's output
	The MLOps landscape in GCP
		Understanding the basic principles of MLOps
		Introducing GCP services related to MLOps
	Exercise – leveraging pre-built GCP models as a service 
		Uploading the image to a GCS bucket
		Creating a detect text function in Python
	Exercise – using GCP in AutoML to train an ML model
	Exercise – deploying a dummy workflow with Vertex AI Pipeline
		Creating a dedicated regional GCS bucket
		Developing the pipeline on Python
		Monitoring the pipeline on the Vertex AI Pipeline console
	Exercise – deploying a scikit-learn model pipeline with Vertex AI
		Creating the first pipeline, which will result in an ML model file in GCS
		Running the first pipeline in Vertex AI Pipeline
		Creating the second pipeline, which will use the model file from the prediction results as a CSV file in GCS
		Running the second pipeline in Vertex AI Pipeline
	Summary 
Section 3: Key Strategies for Architecting Top-Notch Data Pipelines
Chapter 9: User and Project Management in GCP
	Technical requirements 
	Understanding IAM in GCP 
	Planning a GCP project structure
		Understanding the GCP organization, folder, and project hierarchy
		Deciding how many projects we should have in a GCP organization
	Controlling user access to our data warehouse
		Use-case scenario – planninga BigQuery ACL on an e-commerce organization
		Column-level security in BigQuery
	Practicing the concept of IaC using Terraform
		Exercise – creating and running basic Terraform scripts
		Self-exercise – managing a GCP project and resources using Terraform
	Summary
Chapter 10: Cost Strategy in GCP
	Technical requirements
	Estimating the cost of your end-to-end data solution in GCP
		Comparing BigQuery on-demand and flat-rate 
		Example – estimating data engineering use case
	Tips for optimizing BigQuery using partitioned and clustered tables 
		Partitioned tables
		Clustered tables
		Exercise – optimizing BigQuery on-demand cost
	Summary
Chapter 11: CI/CD on Google Cloud Platform for Data Engineers
	Technical requirements
	Introduction to CI/CD
		Understanding the data engineer's relationship with CI/CD practices
	Understanding CI/CD components with GCP services
	Exercise – implementing continuous integration using Cloud Build
		Creating a GitHub repository using Cloud Source Repository
		Developing the code and Cloud Build scripts
		Creating the Cloud Build Trigger
		Pushing the code to the GitHub repository
	Exercise – deploying Cloud Composer jobs using Cloud Build
		Preparing the CI/CD environment
		Preparing the cloudbuild.yaml configuration file
		Pushing the DAG to our GitHub repository
		Checking the CI/CD result in the GCS bucket and Cloud Composer
	Summary
	Further reading
Chapter 12: Boosting Your Confidence as a Data Engineer
	Overviewing the Google Cloud certification
		Exam preparation tips
		Extra GCP services material
	Quiz – reviewing all the concepts you've learned about
		Questions
		Answers
	The past, present, and future of Data Engineering
	Boosting your confidence and final thoughts
	Summary
Index
Other Books You May Enjoy