Big Data Analytics: Theory, Techniques, Platforms, and Applications (SpringerBriefs in Applied Sciences and Technology)

Foreword
Preface
Contents
List of€Figures
1 Introduction
	1.1 Essential Big Data Analytics Properties
	1.2 Big Data Analytics Techniques
	1.3 Overview of This Book
2 Big Data
	2.1 Definition of Big Data
	2.2 Characteristics of Big Data
	2.3 The 5 Vs of Big Data
		2.3.1 Volume
		2.3.2 Value
		2.3.3 Variety
		2.3.4 Velocity
		2.3.5 Veracity
	2.4 Challenges in Big Data
		2.4.1 Data Collection and Storage Challenges
		2.4.2 Data Quality and Integrity Challenges
		2.4.3 Privacy and Security Concerns
		2.4.4 Issues with Extracting Value from Big Data
	2.5 Harnessing the Potential of Big Data
		2.5.1 Advanced Analytics and Machine Learning Opportunities
		2.5.2 Data Visualisation and Communication Opportunities
		2.5.3 Future Directions and Emerging Trends
3 Big Data Analytics
	3.1 What Is Big Data Analytics?
	3.2 The Types of Big Data Analytics
		3.2.1 Descriptive Analytics
		3.2.2 Diagnostic Analytics
		3.2.3 Predictive Analytics
		3.2.4 Prescriptive Analytics
		3.2.5 Cognitive Analytics
	3.3 The Advantages of Big Data Analytics
		3.3.1 Risk Management
		3.3.2 Cost Reduction
		3.3.3 Advanced Data-Driven Decision-Making
		3.3.4 Improving New Product Development
	3.4 The Challenges of Big Data Analytics
		3.4.1 Lack of Knowledge Professionals
		3.4.2 Misunderstanding of Big Data
		3.4.3 Data Growth Issues
		3.4.4 Confusion on Big Data Tool Selection
		3.4.5 Data Security and Privacy
	3.5 The Steps of Big Data Analytics
		3.5.1 Big Data Acquisition
		3.5.2 Big Data Preprocessing
		3.5.3 Big Data Storage
		3.5.4 Big Data Analysis
4 Cloud Computing for Big Data Analytics
	4.1 What is Cloud Computing?
	4.2 The History of Cloud Computing
		4.2.1 Computing Generations
	4.3 Cloud Computing Units
		4.3.1 Cloud Computing Service Models
		4.3.2 Cloud Computing Deployment Models
	4.4 Multi-Cloud Strategies in Big Data Analytics
	4.5 Cloud Computing Platforms for Big Data Analytics
		4.5.1 Amazon Web Services (AWS)
		4.5.2 Microsoft Azure
		4.5.3 Google Cloud Platform (GCP)
		4.5.4 Comparison of Cloud Computing Providers
	4.6 Learning Outcomes of the Chapter
5 Big Data Analytics Platforms
	5.1 Main Characteristics of Big Data Analytics Platforms
		5.1.1 Distributed Computing
		5.1.2 Data Ingestion and Integration
		5.1.3 Data Storage and Management
		5.1.4 Data Processing and Analysis
		5.1.5 Machine Learning and Advanced Analytics
		5.1.6 Data Visualisation and Reporting
		5.1.7 Scalability and Performance
		5.1.8 Security and Governance
	5.2 Desired Properties of a Big Data System
		5.2.1 Robustness and Fault Tolerance
		5.2.2 Scalability
			5.2.2.1 Scaling Solutions for Big Data
		5.2.3 Generalisation
		5.2.4 Extensibility
		5.2.5 Low Latency Reads and Updates
		5.2.6 Minimal Maintenance
		5.2.7 Debuggability
	5.3 Big Data Processing Systems
	5.4 Big Data Processing with Hadoop
		5.4.1 MapReduce Paradigm
		5.4.2 Hadoop Distributed File System (HDFS)
		5.4.3 Yet Another Resource Negotiator (YARN)
		5.4.4 Installing Multi-node Hadoop Cluster
			5.4.4.1 Prerequisites
			5.4.4.2 Downloading and Setting Values
			5.4.4.3 Setting Up a Multi-node Cluster
			5.4.4.4 Starting the Cluster
	5.5 Apache Spark for Big Data Processing
		5.5.1 Apache Spark Core
			5.5.1.1 MLlib for Machine Learning
			5.5.1.2 Spark Streaming for Real-Time Data Processing
			5.5.1.3 Spark SQL for Interactive Queries
			5.5.1.4 GraphX for Graph Processing
		5.5.2 Deploying Spark on YARN
			5.5.2.1 Prerequisites
			5.5.2.2 Installation
			5.5.2.3 Integrate Spark with YARN
		5.5.3 Case Study
	5.6 Apache Hive for Data Engineering
		5.6.1 Deploying Hive on YARN
		5.6.2 Installation
		5.6.3 Integration of Hive with Hadoop YARN
		5.6.4 Case Study
	5.7 Apache Sqoop for Data Ingestion
		5.7.1 Installation
		5.7.2 Configuration of Apache Sqoop
		5.7.3 Case Study
	5.8 Streaming Data Ingestion with Apache Flume
		5.8.1 Installation
		5.8.2 Configuration of Apache Flume and Case Study
	5.9 Apache Mahout: Distributed Machine Learning for Big Data Analytics
		5.9.1 Installation and Configuration of Apache Mahout
		5.9.2 Case Study
	5.10 Learning Outcomes of the Chapter
6 Big Data Storage Solutions
	6.1 Importance of Storage Systems for Big Data
	6.2 Traditional Storage Systems for Big Data
		6.2.1 Relational Databases
		6.2.2 Data Warehouses
		6.2.3 Network Attached Storage (NAS)
		6.2.4 Storage Area Networks (SAN)
	6.3 Big Data Storage Solutions
		6.3.1 Hadoop Distributed File System (HDFS)
		6.3.2 NoSQL Databases
		6.3.3 Cloud Storage Solutions
		6.3.4 Object Storage Systems
		6.3.5 In-Memory Databases
	6.4 Choosing the Right Big Data Storage Solution
		6.4.1 Factors to Consider
		6.4.2 Scalability and Performance Requirements
	6.5 Future Trends in Big Data Storage
		6.5.1 Advances in Storage Technologies
		6.5.2 Edge Computing and Distributed Storage
		6.5.3 AI and Machine Learning in Storage
	6.6 Learning Outcomes of the Chapter
7 Big Data Monitoring
	7.1 Understanding Monitoring
	7.2 Identifying the Types of Monitoring
		7.2.1 Proactive Monitoring
		7.2.2 Reactive Monitoring
	7.3 The Need for Monitoring
	7.4 The Components of Monitoring
		7.4.1 Alerts/Notifications
		7.4.2 Events
		7.4.3 Logs
		7.4.4 Metrics
		7.4.5 Incidence
		7.4.6 Debugging Ability
	7.5 Available Monitoring Tools for Big Data Systems
		7.5.1 DataDog
		7.5.2 SequenceIQ
		7.5.3 Sematext
		7.5.4 Apache Chukwa
		7.5.5 Nagios
		7.5.6 Ganglia
		7.5.7 DMon
		7.5.8 SmartMonit
	7.6 Learning Outcomes of the Chapter
8 Debugging Big Data Systems  for Big Data Analytics
	8.1 Debugging for Real-World Performance Problems
	8.2 Debugging Steps
	8.3 Problems in Big Data Systems
		8.3.1 Data Locality
		8.3.2 Resource Heterogeneity
		8.3.3 Network Issues
		8.3.4 Resource Over-Allocation
		8.3.5 Unnecessary Speculation
		8.3.6 Poor Scheduling Policy
	8.4 Root Cause Analysis in Big Data Systems
		8.4.1 Importance of Root Cause Analysis in Big Data Analytics
		8.4.2 Root Cause Analysis Steps
		8.4.3 Tools and Techniques for RCA in Big Data Systems
		8.4.4 Challenges and Considerations in RCA for Big Data Systems
	8.5 Available Diagnosis Tools for Big Data Systems
		8.5.1 Mantri
		8.5.2 TACC Stats
		8.5.3 DCDB Wintermute
		8.5.4 AutoDiagn
	8.6 Learning Outcomes of the Chapter
9 Machine Learning for Big Data Analytics
	9.1 Harnessing Machine Learning for Big Data Insights
	9.2 Supervised Machine Learning for Big Data Analytics
		9.2.1 Challenges of Applying Supervised Machine Learning to Big Data Analytics
		9.2.2 Pre-processing Big Data for Supervised Machine Learning
		9.2.3 Popular Supervised Machine Learning Algorithms for Big Data Analytics
	9.3 Unsupervised Machine Learning for Big Data Analytics
		9.3.1 K-means Clustering
		9.3.2 Hierarchical Clustering
		9.3.3 DBSCAN
		9.3.4 Gaussian Mixture Models (GMM)
		9.3.5 Principal Component Analysis (PCA)
		9.3.6 t-SNE
		9.3.7 Apriori Algorithm
		9.3.8 Isolation Forest
		9.3.9 Expectation-Maximisation Algorithm
		9.3.10 Spectral Clustering
		9.3.11 Mean Shift
	9.4 Neural Networks Algorithms
		9.4.1 The Components of Neural Networks
		9.4.2 The Types of Neural Networks
	9.5 Probabilistic Learning for Big Data Analytics
		9.5.1 Fundamentals of Probabilistic Learning
		9.5.2 Scalable Algorithms for Probabilistic Learning
		9.5.3 Applications of Probabilistic Learning in Big Data Analytics
	9.6 Performance Evaluation and Optimisation Techniques
		9.6.1 Evaluation Metrics for Supervised Machine Learning Algorithms
		9.6.2 Cross-Validation Techniques
		9.6.3 Hyperparameter Optimisation Techniques
	9.7 Learning Outcomes of the Chapter
10 Real-World Big Data Analytics Case Studies
	10.1 Government Sector
		10.1.1 Enhancing Public Services Through Data-Driven Governance
		10.1.2 Predictive Analytics for Smart City Planning
		10.1.3 Security and Surveillance: Big Data in Government
		10.1.4 Election Forecasting and Voter Analytics
	10.2 Healthcare Industry
		10.2.1 Revolutionising Healthcare with Big Data Analytics
		10.2.2 Precision Medicine: Tailoring Treatments with Data
		10.2.3 Disease Outbreak Prediction and Prevention
	10.3 Entertainment Industry
		10.3.1 Content Personalization and Recommendation Systems
		10.3.2 Box Office Predictions and Revenue Optimization
		10.3.3 Audience Engagement and Social Media Analytics
	10.4 Banking Sector
		10.4.1 Risk Assessment and Credit Scoring
		10.4.2 Customer Relationship Management (CRM)  and Personalization
		10.4.3 Fraud Detection and Security
		10.4.4 Strategic Decision-Making and Regulatory Compliance
	10.5 Retail Industry
		10.5.1 Inventory Management and Demand Forecasting
		10.5.2 Customer Segmentation and Personalization
		10.5.3 Supply Chain Optimization and Vendor Management
		10.5.4 Enhanced Customer Experience Through In-Store Analytics
	10.6 Energy and Utilities
		10.6.1 Grid Management and Smart Grids
		10.6.2 Predictive Maintenance and Asset Optimization
		10.6.3 Energy Generation and Renewable Integration
		10.6.4 Energy Efficiency and Demand Response
		10.6.5 Environmental Sustainability and Emissions Reduction
	10.7 Learning Outcomes of the Chapter
11 Big Data Analytics in Smart Grids
	11.1 Smart Grids
	11.2 Big Data Analytics in Smart Grid
		11.2.1 Need of Big Data Analytics for Smart Grids
		11.2.2 Big Data and Cloud Computing
	11.3 Example of Big Data Analytics in Smart Grid
		11.3.1 Data Pre-processing
		11.3.2 Machine Learning Models
		11.3.3 Results and Evaluations
	11.4 Learning Outcomes of the Chapter
12 Big Data Analytics in Bioinformatics
	12.1 Big Data: Bioinformatic Perspective
		12.1.1 Big Data Problems in Bioinformatics
	12.2 Frameworks for Big Genome Data
	12.3 Biological Databases
	12.4 Big Data Analytics in Bioinformatics
		12.4.1 Hadoop and MapReduce in Bioinformatics Analytics
		12.4.2 Bioinformatics Pipelines and Workflows for Big Data
		12.4.3 Analysis Pipelines and Tools with Hadoop (MapReduce) Framework
		12.4.4 Deep Learning in Bioinformatics
	12.5 Variant Detection in Genome:A Case Study
		12.5.1 Genom Data Copying to HDFS
		12.5.2 Big Genome Data Processing Using MapReduce
	12.6 Learning Outcomes of the Chapter