Scalable Data Streaming with Amazon Kinesis: Design and secure highly available, cost-effective data streaming applications with Amazon Kinesis

Cover
Title Page
Copyright and Credits
Contributors
Table of Contents
Preface
Section 1: Introduction to Data Streaming and Amazon Kinesis
Chapter 1: What Are Data Streams?
	Introducing data streams
		Sources of data
	The value of real-time data in analytics
	Decoupling systems
	Challenges associated with distributed systems
		Transactions per second
		Scaling
		Latency
		Fault tolerance/high availability
	Overview of messaging concepts
		Overview of core messaging components
		Messaging concepts
	Examples of data streaming
		Application log processing
		Internet of Things
		Real-time recommendations
		Video streams
	Summary
	Further reading
Chapter 2: Messaging and Data Streaming in AWS
	Amazon Kinesis Data Streams (KDS)
		Encryption, authentication, and authorization
		Producing and consuming records
		Data delivery guarantees
		Integration with other AWS services
		Monitoring
	Amazon Kinesis Data Firehose (KDF)
		Encryption, authentication, and authorization
		Monitoring
		Producers
		Delivery destinations
		Transformations
	Amazon Kinesis Data Analytics (KDA)
		Amazon KDA for SQL
		Amazon Kinesis Data Analytics for Apache Flink (KDA Flink)
	Amazon Kinesis Video Streams (KVS)
	Amazon Simple Queue Service (SQS)
	Amazon Simple Notification Service (SNS)
		Amazon SNS integrations with other AWS services
		Encryption at rest
	Amazon MQ for Apache ActiveMQ
	IoT Core
		Device software
		Control services
		Analytics services
	Amazon Managed Streaming for Apache Kafka (MSK)
		Apache Kafka
		Amazon MSK
	Amazon EventBridge
	Service comparison summary
	Summary
Chapter 3: The SmartCity Bike-Sharing Service
	The mission for sustainable transportation
	SmartCity new mobile features
		SmartCity data pipeline
		SmartCity data lake
		SmartCity operations and analytics dashboard
		SmartCity video
	The AWS Well-Architected Framework
	Summary
	Further reading
Section 2: Deep Dive into Kinesis
Chapter 4: Kinesis Data Streams
	Technical requirements
	Discovering Amazon Kinesis Data Streams
		Creating streams and shards
	Creating a stream producer application
	Creating a stream consumer application
	Data pipelines with Amazon Kinesis Data Streams
		Data pipeline design (simple)
		Data pipeline design (intermediate)
		Data pipeline design (full design)
		Designing for scalable and reliable analytics pipelines
		Monitoring and scaling with Amazon Kinesis Data Streams
		X-Ray tracing with Amazon Kinesis Data Streams
		Scaling up with Amazon Kinesis Data Streams
		Securing Amazon Kinesis Data Streams
		Implementing least-privilege access
	Summary
	Further reading
Chapter 5: Kinesis Firehose
	Technical requirements
		Setting up the AWS account
		Using a local development environment
		Using an AWS Cloud9 development environment
		Code examples
	Discovering Amazon Kinesis Firehose
		Understanding KDF delivery streams
	Understanding encryption in KDF
	Using data transformation in KDF with a Lambda function
	Understanding delivery stream destinations
		Amazon S3
		Amazon Redshift
		Amazon Elasticsearch Service
		Splunk destination
		HTTP endpoint destination
	Understanding data format conversion in KDF
		Deserialization
		Schema
		Serializer
		Data format conversion errors
	Understanding monitoring in KDF
	Use-case example – Bikeshare station data pipeline with KDF
		Steps to recreate the example
	Summary
	Further reading
Chapter 6: Kinesis Data Analytics
	Technical requirements
		AWS account setup
		AWS CDK
		Java and Java IDE
		Code examples
	Discovering Amazon KDA
	Working on SmartCity bike share analytics use cases
	Creating operational insights using SQL Engine
		Core concepts and capabilities
	Creating operational insights using Apache Flink
		Options for running Flink applications in AWS Cloud
		Flink applications on KDA
	Building bike ride analytic applications
		Setting up a producer application
		Building a KDA SQL application
		Building a KDA Flink application
	Monitoring KDA applications
	Summary
	Further reading
		Blogs
		Workshops
Chapter 7: Amazon Kinesis Video Streams
	Technical requirements
		AWS account setup
		Using a local development environment
		Code examples
	Understanding video fundamentals
		Containers
		Codecs
	Discovering Amazon Kinesis video streams WebRTC
		Core concepts and connection patterns
		Creating a signaling channel
		Establishing a connection
	Discovering Amazon KVS
		Key components of KVS
		Stream
		Kinesis producer
		Consuming
		Creating a stream
		Producing
		Integration with Rekognition
	Building video-enabled applications with KVS
	Summary
	Further reading
Section 3: Integrations
Chapter 8: Kinesis Integrations
	Technical requirements
		AWS account setup
		AWS CLI
		Kinesis Data Generator
		Code examples
	Amazon services that can produce data to send to Kinesis
		Amazon Connect
		Amazon Aurora database activity
		DynamoDB activity
		Processing Kinesis data with Apache Spark
		Amazon services that consume data from Kinesis
		Serverless data lake
	Amazon services that transform Kinesis data
		Routing events with EventBridge
	Third-party integrations with Kinesis
		Splunk
	Summary
	Further reading
	Why subscribe?
Other Books You May Enjoy
Index