Data Analytics in the AWS Cloud: Building a Data Platform for BI and Predictive Analytics on AWS

Cover
Title Page
Copyright Page
About the Author
About the Technical Editor
Acknowledgments
Contents at a Glance
Contents
Introduction
	What Is a Data Lake?
		When You Do Not Need a Data Lake
		When Do You Need Analytics?
		When Do You Need a Data Lake for Analytics?
		How About an Analytics Team?
	The Data Platform
	The End of the Beginning
Chapter 1 AWS Data Lakes and Analytics Technology Overview
	Why AWS?
	What Does a Data Lake Look Like in AWS?
	Analytics on AWS
	Skills Required to Build and Maintain an AWS Analytics Pipeline
Chapter 2 The Path to Analytics: Setting Up a Data and Analytics Team
	The Data Vision
		Support
	DA Team Roles
		Early Stage Roles
			Team Lead
			Data Architect
			Data Engineer
			Data Analyst
		Maturity Stage Roles
			Data Scientist
			Cloud Engineer
			Business Intelligence (BI) Developer
			Machine Learning Engineer
			Business Analyst
		Niche Roles
	Analytics Flow at a Process Level
		Workflow Methodology
	The DA Team Mantra: “Automate Everything”
	Analytics Models in the Wild: Centralized, Distributed, Center of Excellence
		Centralized
		Distributed
		Center of Excellence
	Summary
Chapter 3 Working on AWS
	Accessing AWS
	Everything Is a Resource
		S3: An Important Exception
	IAM: Policies, Roles, and Users
		Policies
			Identity-Based Policies
			Resource-Based Policies
			Roles
			Users and User Groups
		Summarizing IAM
	Working with the Web Console
	The AWS Command-Line Interface
		Installing AWS CLI
			Linux Installation
			macOS Installation
			Windows
		Configuring AWS CLI
			A Note on Region
			Setting Individual Parameters
			Using Profiles and Configuration Files
			Final Notes on Configuration
		Using the AWS CLI
		Using Skeletons and File Inputs
		Cleaning Up!
	Infrastructure-as-Code: CloudFormation and Terraform
		CloudFormation
			CloudFormation Stacks
			CloudFormation Template Anatomy
			CloudFormation Changesets
			Getting Stack Information
			Cleaning Up Again
			CloudFormation Conclusions
		Terraform
			Coding Style
			Modularity
			Limitations
		Terraform vs. CloudFormation
			Infrastructure-as-Code: CDK, Pulumi, Cloudcraft, and Other Solutions
			AWS CDK
			Pulumi
			Cloudcraft
		Infrastructure Management Conclusions
Chapter 4 Serverless Computing and Data Engineering
	Serverless vs. Fully Managed
	AWS Serverless Technologies
		AWS Lambda
			Pricing Model
			Laser Focus on Code
			The Lambda Paradigm Shift
			Virtually Infinite Scalability
			Geographical Distribution
			A Lambda Hello World
			Lambda Configuration
			Runtime
			Container-Based Lambdas
			Architectures
			Memory
			Networking
			Execution Role
			Environment Variables
		AWS EventBridge
		AWS Fargate
		AWS DynamoDB
		AWS SNS
		Amazon SQS
		AWS CloudWatch
		Amazon QuickSight
		AWS Step Functions
		Amazon API Gateway
		Amazon Cognito
	AWS Serverless Application Model (SAM)
		Ephemeral Infrastructure
		AWS SAM Installation
		Configuration
		Creating Your First AWS SAM Project
		Application Structure
		SAM Resource Types
		SAM Lambda Template
		!! Recursive Lambda Invocation !!
		Function Metadata
		Outputs
		Implicitly Generated Resources
		Other Template Sections
		Lambda Code
		Building Your First SAM Application
		Testing the AWS SAM Application Locally
		Deployment
		Cleaning Up
	Summary
Chapter 5 Data Ingestion
	AWS Data Lake Architecture
		Serverless Data Lake Architecture Structure
			Ingestion
			Storage and Processing
			Cataloging, Governance, and Search
			Security and Monitoring
			Consumption
	Sample Processing Architecture: Cataloging Images into DynamoDB
		Use Case Description
		SAM Application Creation
			S3-Triggered Lambda
		Adding DynamoDB
		Lambda Execution Context
		Inserting into DynamoDB
		Cleaning Up
	Serverless Ingestion
		AWS Fargate
		AWS Lambda
		Example Architecture: Fargate-Based Periodic Batch Import
			The Basic Importer
			ECS CLI
			AWS Copilot CLI
			Clean Up
		AWS Kinesis Ingestion
			Example Architecture: Two-Pronged Delivery
	Fully Managed Ingestion with AppFlow
	Operational Data Ingestion with Database Migration Service
		DMS Concepts
			DMS Instance
			DMS Endpoints
			DMS Tasks
			Summary of the Workflow
		Common Use of DMS
		Example Architecture: DMS to S3
			DMS Instance
			DMS Endpoints
			DMS Task
	Summary
Chapter 6 Processing Data
	Phases of Data Preparation
		What Is ETL? Why Should I Care?
		ETL Job vs. Streaming Job
	Overview of ETL in AWS
		ETL with AWS Glue
		ETL with Lambda Functions
		ETL with Hadoop/EMR
		Other Ways to Perform ETL
	ETL Job Design Concepts
		Source Identification
		Destination Identification
		Mappings
		Validation
		Filter
		Join, Denormalization, Relationalization
	AWS Glue for ETL
		Really, It’s Just Spark
		Visual
		Spark Script Editor
		Python Shell Script Editor
		Jupyter Notebook
	Connectors
		Creating Connections
			Creating Connections with the Web Console
			Creating Connections with the AWS CLI
	Creating ETL Jobs with AWS Glue Visual Editor
		ETL Example: Format Switch from Raw (JSON) to Cleaned (Parquet)
		Job Bookmarks
		Transformations
			Apply Mapping
			Filter
			Other Available Transforms
			Run the Edited Job
		Visual Editor with Source and Target Conclusions
	Creating ETL Jobs with AWS Glue Visual Editor (without Source and Target)
	Creating ETL Jobs with the Spark Script Editor
	Developing ETL Jobs with AWS Glue Notebooks
		What Is a Notebook?
		Notebook Structure
		Step 1: Load Code into a DynamicFrame
		Step 2: Apply Field Mapping
		Step 3: Apply the Filter
		Step 4: Write to S3 in Parquet Format
		Example: Joining and Denormalizing Data from Two S3 Locations
		Conclusions for Manually Authored Jobs with Notebooks
	Creating ETL Jobs with AWS Glue Interactive Sessions
		It’s Magic
		Development Workflow
	Streaming Jobs
		Differences with a Standard ETL Job
		Streaming Sources
		Example: Process Kinesis Streams with a Streaming Job
		Streaming ETL Jobs Conclusions
		Summary
Chapter 7 Cataloging, Governance, and Search
	Cataloging with AWS Glue
		AWS Glue and the AWS Glue Data Catalog
		Glue Databases and Tables
			Databases
			The Idea of Schema-on-Read
			Tables
			Create Table Manually
			Creating a Table from an Existing Schema
			Creating a Table with a Crawler
			Summary on Databases and Tables
			Crawlers
			Updating or Not Updating?
			Running the Crawler
			Creating a Crawler from the AWS CLI
			Retrieving Table Information from the CLI
		Classifiers
			Classifier Example
		Crawlers and Classifiers Summary
	Search with Amazon Athena: The Heart of Analytics in AWS
		A Bit of History
		Interface Overview
		Creating Tables Manually
		Athena Data Types
			Complex Types
		Running a Query
		Connecting with JDBC and ODBC
		Query Stats
		Recent Queries and Saved Queries
		The Power of Partitions
			Athena Pricing Model
			Automatic Naming
		Athena Query Output
		Athena Peculiarities (SQL and Not)
			Computed Fields Gotcha and WITH Statement Workaround
			Lowercase!
			Query Explain
			Deduplicating Records
			Working with JSON, Flattening, and Unnesting
		Athena Views
		CREATE TABLE AS SELECT (CTAS)
		Saving Queries and Reusing Saved Queries
		Running Parameterized Queries
		Athena Federated Queries
			Athena Lambda Connectors
			Note on Connection Errors
		Performing Federated Queries
		Creating a View from a Federated Query
	Governing: Athena Workgroups, Lake Formation, and More
		Athena Workgroups
		Fine-Grained Athena Access with IAM
		Recap of Athena-Based Governance
	AWS Lake Formation
		Registering a Location in Lake Formation
		Creating a Database in Lake Formation
			Assigning Permissions in Lake Formation
		LF-Tags and Permissions in Lake Formation
		Data Filters
		Governance Conclusions
	Summary
Chapter 8 Data Consumption: BI, Visualization, and Reporting
	QuickSight
		Signing Up for QuickSight
			Standard Plan
			Enterprise Plan
		Users and User Groups
			Managing Users and Groups
		Managing QuickSight
			Users and Groups
			Your Subscriptions
			SPICE Capacity
			Account Settings
			Security and Permissions
			VPC Connections
			Mobile Settings
			Domains and Embedding
			Single Sign-On
		Data Sources and Datasets
			Creating an Athena Data Source
			Creating Other Data Sources
			Creating a Data Source from the AWS CLI
			Creating a Dataset from a Table
			Creating a Dataset from a SQL Query
			Duplicating Datasets
			Note on Creating Datasets
		QuickSight Favorites, Recent, and Folders
		SPICE
			Manage SPICE Capacity
			Refresh Schedule
		QuickSight Data Editor
			QuickSight Data Types
			Change Data Types
			Calculated Fields
			Joining Data
			Excluding Fields
			Filtering Data
			Removing Data
			Geospatial Hierarchies and Adding Fields to Hierarchies
			Unsupported Format Dates
		Visualizing Data: QuickSight Analysis
			Adding a Title and a Description to Your Analysis
			Renaming the Sheet
			Your First Visual with AutoGraph
			Field Wells
			Visual Types
			Saving and Autosaving
			A First Example: Pie Chart
			Renaming a Visual
			Filtering Data
			Adding Drill-Downs
			Parameters
			Actions
			Insights
			ML-Powered Insights
			Sharing an Analysis
		Dashboards
			Dashboard Layouts and Themes
			Publishing a Dashboard
			Embedding Visuals and Dashboards
	Data Consumption: Not Only Dashboards
	Summary
Chapter 9 Machine Learning at Scale
	Machine Learning and Artificial Intelligence
		What Are ML/AI Use Cases?
		Types of ML Models
		Overview of ML/AI AWS Solutions
	Amazon SageMaker
		SageMaker Domains
			Adding a User to the Domain
		SageMaker Studio
		SageMaker Example Notebook
			Step 1: Prerequisites and Preprocessing
			Step 2: Data Ingestion
			Step 3: Data Inspection
			Step 4: Data Conversion
			Step 5: Upload Training Data
			Step 6: Train the Model
			Step 7: Set Up Hosting and Deploy the Model
			Step 8: Validate the Model
			Step 9: Use the Model
		Inference
			Real Time
			Asynchronous
			Serverless
			Batch Transform
		Data Wrangler
		SageMaker Canvas
	Summary
Appendix Example Data Architectures in AWS
	Modern Data Lake Architecture
		ETL in a Lake House
		Consuming Data in the Lake House
		The Modern Data Lake Architecture
	Batch Processing
	Stream Processing
	Architecture Design Recommendations
		Automate Everything
		Build on Events
		Performance = Cost Savings
		AWS Glue Catalog and Athena-Centric Workflow
		Design Flexible
		Pick Your Battles
		Parquet
	Summary
Index
EULA