Scalable Data Architecture with Java: Build efficient enterprise-grade data architecting solutions using Java

Cover
Title Page
Copyright and Credits
Contributors
About the reviewers
Table of Contents
Preface
Section 1 – Foundation of Data Systems
Chapter 1: Basics of Modern Data Architecture
	Exploring the landscape of data engineering
		What is data engineering?
		Dimensions of data
		Types of data engineering problems
	Responsibilities and challenges of a Java data architect
		Data architect versus data engineer
		Challenges of a data architect
	Techniques to mitigate those challenges
	Summary
Chapter 2: Data Storage and Databases
	Understanding data types, formats, and encodings
		Data types
		Data formats
	Understanding file, block, and object storage
		File storage
		Block storage
		Object storage
	The data lake, data warehouse, and data mart
		Data lake
		Data warehouse
		Data marts
	Databases and their types
		Relational database
		NoSQL database
	Data model design considerations
	Summary
Chapter 3: Identifying the Right Data Platform
	Technical requirements
	Virtualization and containerization platforms
		Benefits of virtualization
		Containerization
		Benefits of containerization
		Kubernetes
	Hadoop platforms
		Hadoop architecture
	Cloud platforms
		Benefits of cloud computing
	Choosing the correct platform
		When to choose virtualization versus containerization
		When to use big data
		Choosing between on-premise versus cloud-based solutions
		Choosing between various cloud vendors
	Summary
Section 2 – Building Data Processing Pipelines
Chapter 4: ETL Data Load – A Batch-Based Solution to Ingesting Data in a Data Warehouse
	Technical requirements
	Understanding the problem and source data
		Problem statement
		Understanding the source data
	Building an effective data model
		Relational data warehouse schemas
		Evaluation of the schema design
	Designing the solution
	Implementing and unit testing the solution
	Summary
Chapter 5: Architecting a Batch Processing Pipeline
	Technical requirements
	Developing the architecture and choosing the right tools  
		Problem statement
		Analyzing the problem
		Architecting the solution
		Factors that affect your choice of storage
		Determining storage based on cost
		The cost factor in the processing layer
	Implementing the solution
		Profiling the source data
		Writing the Spark application
		Deploying and running the Spark application
		Developing and testing a Lambda trigger
		Performance tuning a Spark job
	Querying the ODL using AWS Athena
	Summary
Chapter 6: Architecting a Real-Time Processing Pipeline
	Technical requirements
	Understanding and analyzing the streaming problem
		Problem statement
		Analyzing the problem
	Architecting the solution
	Implementing and verifying the design
		Setting up Apache Kafka on your local machine
		Developing the Kafka streaming application
		Unit testing a Kafka Streams application
		Configuring and running the application
		Creating a MongoDB Atlas cloud instance and database
		Configuring Kafka Connect to store the results in MongoDB
		Verifying the solution
	Summary
Chapter 7: Core Architectural Design Patterns
	Core batch processing patterns
		The staged Collect-Process-Store pattern
		Common file format processing pattern
		The Extract-Load-Transform pattern
		The compaction pattern
		The staged report generation pattern
	Core stream processing patterns
		The outbox pattern
		The saga pattern
		The choreography pattern
		The Command Query Responsibility Segregation (CQRS) pattern  
		The strangler fig pattern  
		The log stream analytics pattern
	Hybrid data processing patterns
		The Lambda architecture  
		The Kappa architecture
	Serverless patterns for data ingestion
	Summary
Chapter 8: Enabling Data Security and Governance
	Technical requirements
	Introducing data governance – what and why
		When to consider data governance
		The DGI data governance framework
	Practical data governance using DataHub and NiFi
		Creating the NiFi pipeline
		Setting up DataHub
		Governance activities
	Understanding the need for data security
	Solution and tools available for data security
	Summary
Section 3 – Enabling Dataas a Service
Chapter 9: Exposing MongoDB Data as a Service
	Technical requirements
	Introducing DaaS – what and why
		Benefits of using DaaS
	Creating a DaaS to expose data using Spring Boot
		Problem statement
		Analyzing and designing a solution
		Implementing the Spring Boot REST application
		Deploying the application in an ECS cluster
	API management
	Enabling API management over the DaaS API using AWS API Gateway
	Summary
Chapter 10: Federated and Scalable DaaS with GraphQL
	Technical requirements
	Introducing GraphQL – what, when, and why
		Operation types
		Why use GraphQL?
		When to use GraphQL
	Core architectural patterns of GraphQL
	A practical use case – exposing federated data models using GraphQL
	Summary
Section 4 – Choosing Suitable Data Architecture
Chapter 11: Measuring Performance and Benchmarking Your Applications
	Performance engineering and planning
		Performance engineering versus performance testing
	Tools for performance engineering
	Publishing performance benchmarks
	Optimizing performance
		Java Virtual Machine and garbage collection optimizations
		Big data performance tuning
		Optimizing streaming applications
		Database tuning
	Summary
Chapter 12: Evaluating, Recommending, and Presenting Your Solutions
	Creating cost and resource estimations
		Storage and compute capacity planning
		Effort and timeline estimation
	Creating an architectural decision matrix
	Data-driven architectural decisions to mitigate risk
	Presenting the solution and recommendations
	Summary
Index
Other Books You May Enjoy