Practical Machine Learning with Spark: Uncover Apache Spark’s Scalable Performance with High-Quality Algorithms Across NLP, Computer Vision and ML(English Edition)

Cover Page
Title Page
Copyright Page
Dedication Page
About the Authors
About the Reviewers
Acknowledgements
Preface
Errata
Table of Contents
1. Introduction to Machine Learning
	Introduction
	Structure
	Objectives
	Evolution of Machine Learning
	Fundamentals and Definition of Machine Learning
	Types of Machine Learning
	Learning of Models Based on the First Criteria
		Supervised Learning (SL)
		Unsupervised Learning (USL)
		Reinforcement Learning (RL)
		Hybrid Learning Problem (HLP)
	Learning of Models Based on Second Criteria (Batch Mode Learning and Online Mode Learning)
		Batch Learning
		Online Learning
	Applications of Machine Learning
		Recommendation Engine
		Financial Services
		Social Media
		Face Recognition
		Healthcare
		Sentiment Analysis
		Video Surveillance
	Future Scope of Machine Learning
		A New Trail of Intelligence Augmentation (IA)
		Edge Computing with ML
		Quantum Computing with ML
		Improved Cognitive Services
		Robotics
		Machine Learning in Space Exploration
		Self-driving Cars and Autonomous Transportation
		Enhanced Healthcare using AI
	Conclusion
2. Apache Spark Environment Setup and Configuration
	Introduction
	Structure
	Objectives
	Laconic View on Apache Spark
	Apache Spark Installation using Hortonworks Sandbox
		VMware Workstation Player Installation
		ClouderaVM Installation for HDP
	Apache Hadoop and Apache Spark Setup on Amazon Web Services (AWS)
	AWS Account Credentials and Amazon EC2 Creation
		PuTTY and PuTTYgen Software for Generating a .ppk file from a .pem and Accessing the Amazon EC2 Instance Through a Public IP Address
	Apache Ambari Installation on Amazon EC2
		Disabling the iptables
		Installation of Apache Ambari Repository and Hadoop Services on Amazon EC2
	Python Editors for the Spark Programming Framework
	Sublime Editor
	PySpark or Python Codebase Syncing from a Server to a Local Directory and Vice Versa
	Jupyter Notebook
	Microsoft PowerBI Installation for Data Visualization
	DBeaver Installation for Accessing the Data from the Persistence Layer
	Apache Spark Installation on Google Colab
	Conclusion
3. Apache Spark
	Introduction
	Structure
	Objectives
	Need of Apache Spark
	Evolution of Apache Spark
	Apache Spark Components
	Architecture of Apache Spark
	Resilient Distributed Dataset (RDD)
	Direct Acyclic Graph (DAG) in Spark
	Lazy Evaluation
	DataFrames
	Datasets
	Accumulator and Broadcast
		Accumulator
		Broadcast
	Apache Spark Optimization and its Techniques
	Memory Storage Levels: Cache and Persist
	Spark Submit
	Spark Monitoring
	Apache Livy: An Easy Interaction With a Spark Cluster Over a REST Interface
	Job Scheduling
	Spark RDD Operations: Transformation and Action
	Data Ingestion in Apache Spark
	Application of Apache Spark
	Conclusion
4. Apache Spark MLlib
	Introduction
	Structure
	Objectives
	Spark MLlib Algorithms
		Classification Category
		Regression Category
		Clustering Category
	ML Components/Pipelines
		DataFrame
		Transformer
		Estimator
		Pipeline
		Parameter
		CrossValidator
		Evaluator
	Spark MLlib’s Datatypes
		Local Vector
		Sparse Vector
		DenseVector
		LabelPoint
		Local Matrix
		Distributed Matrix
	Extracting, Transforming, and Selecting Features
		Term Frequency-Inverse Document Frequency (TF-IDF)
		Word2Vec
		CountVectorizer
		FeatureHasher
	Feature Transformers
		Tokenizer
		StopWordsRemover
		N-Gram
		Binarizer
		Principal Component Analysis (PCA)
		Polynomial Expansion
		Discrete Cosine Transform (DCT)
		StringIndexer
		IndexToString
		VectorIndexer
		Normalizer
		StandardScaler
		MinMaxScaler
		MaxAbsScaler
		Bucketizer
		ElementwiseProduct
		SQLTransformer
		VectorAssembler
		VectorSizeHint
		Quantile Discretizer (QD)
		Imputer
	Feature Selectors
		VectorSlicer
		ChiSqSelector
	Conclusion
5. Supervised Learning with Spark
	Introduction
	Structure
	Objectives
	Definition of Supervised Learning
		Regression and its Types
	Regularization in Linear Regression
		Least Absolute Shrinkage and Selection Operator (Lasso Regression)/L1 Regularization
		Ridge Regression/L2 Regularization
		Elastic-net Regression/L1+L2 Regularization
		Generalized Linear Regression (GLR)
		Isotonic Regression/Monotonic Non-Decreasing Regression/ Equal Stretch Regression
	Classification and its Types
	Classification and Regression Tree (CART)
		Terminology in CART
		Decision Tree (DT)
		Decision Tree Classification (DTC) in CART
		Decision Tree Regression (DTR)
		Ensemble Learning (EL)
	Performance Metrics/Evaluation Metrics (EM)
		Classification Metrices
		Regression Metrics
	Churn Prediction Model
	Conclusion
6. Un-Supervised Learning with Apache Spark
	Introduction
	Structure
	Objectives
	Clustering
		K-Means under Clustering
		Bisecting K-means Algorithm (BKM)
		Gaussian Mixture Model (GMM)
		Latent Dirichlet Allocation (LDA)
	Conclusion
7. Natural Language Processing with Apache Spark
	Introduction
	Structure
	Objectives
	Evolution of Natural Language Processing
	NLP and its Types
	Artificial Intelligence-Based Approach
		Deep Learning or The Neural Network Approach
	A Laconic View on SparkNLP
	Advantages of SparkNLP
	Core Execution Blocks of NLP
	Components of NLP
		Morphological Analysis
		Lexical Analysis
		Syntax Analysis
		Semantic Analysis
		Pragmatic Analysis
		Discourse Integration
	Comparison among Natural Language Processing (NLP), Natural Language Understanding (NLU), and Natural Language Generation (NLG)
	Widely Used Libraries of NLP
	Types of NLP
	Features in NLP
	Sentiment Analysis using Spark NLP
	Enhancement in NLP
	Alternate of SparkNLP
	Conclusion
8. Recommendation Engine with Spark
	Introduction
	Structure
	Objectives
	Evolution of a Recommendation Engine
	Types of Recommendation Engines
		Content-Based Filtering (CBF)
		Collaborative Filtering (CF)
		Hybrid Recommendation Engines (HREs)
	Information Collection Phases in RE
		Explicit Feedback
		Implicit Feedback
		Hybrid Feedback
	Real-Time Pipeline of a Recommendation Engine
	Ant Colony Optimization in a Recommendation Engine
	Hidden Markov Chain Model (HMCM)
	Market Basket Algorithm (MBA)
	Implementation of a Recommendation Engine
	Limitations of Recommender Systems
		Cold-Start Problem
	Applications of a Recommendation Engine
	Conclusion
9. Deep Learning with Spark
	Introduction
	Structure
	Objectives
	Evolution of the Neural Network
		Cybernetics
		Connectionism
		Deep Learning (DL)
	Definition of Deep Learning (DL)
		Neural Network and its Model Representations
		Various Terminologies Used in DL
	Feature Engineering (FE)/Feature Selection (FS)
		Filter Method (FM)
		Generalized Method (GM)
		Wrapper Method
		Embedded Method
		Different networks in DL
	Different Activation Functions
		Linear Function or Identity Activation Function (IAF)
		Binary Step Activation Function (BSAF)
		Sigmoid Activation Function/Logistic/Soft Step
		Hyperbolic Tangent Activation Function (HTAF) / Tanh AF
		SoftSign Activation Function
		Swish Activation Function
		Rectified Linear Unit Activation Function (RLUAF) / ReLU / Maximum Function
		Leaky Rectified Linear Unit (Leaky ReLU)
		Parametric Rectified Linear Unit Activation Function (PRLUAF)
		Exponential Linear Unit Activation Function (ELUAF)
		SoftPlus Activation Function (SPAF)
		SoftMax Activation Function (SMAF)
		Scaled Exponential Linear Unit Activation Function (SELUAF)
	Different Types of Loss Functions
		Regression Loss Function
			Mean Square Error Loss (MSEL)/ L2 Loss
			Root Mean Square Error Loss (RMSEL)
			Mean Absolute Error Loss (MAEL)/ L1 loss
			Mean Squared Logarithmic Error (MSLE)
			Mean Absolute Percentage Error Loss (MAPEL)/ Mean Absolute Percentage Deviation Loss (MAPDL)
			Mean Bias Error Loss (MBEL)
			Huber Loss (HL) / Smooth Mean Absolute Error Loss
			LogCosh Loss
		Classification Loss Function
			Hinge Loss/Multi Class SVM Loss
			Squared Hinge Loss Function (SHLF)
			Categorical Hinge Loss Function(CHF)
			Cross Entropy Loss (CEL)/Negative Log Likelihood
			Binary Cross Entropy Loss (BCEL)
			Categorical Cross Entropy Loss (CCEL)
			Kullback Leibler Divergence Loss (KLDL)/ Relative Entropy
			Sparse Categorical Cross Entropy Loss (SCCEL)
			Focal Loss (FL)
	Different Optimizers
		Gradient Descent (GD)
		Batch Gradient Descent (BGD)
		Stochastic Gradient Descent (SGD)/full batch gradient descent
		Mini Batch Gradient Descent (MBGD)
		Momentum Based Gradient Descent (MBGD)
		Nesterov Accelerated Gradient (NAG)
		Adaptive Gradient (Adagrad)
		Adaptive Moment Estimation (Adam)
		AdaDelta
	Cloud notebooks for ML and DL
		Google Colab
	Deep Learning Frameworks
		TensorFlow
		PyTorch
		Keras
		Caffe
		MxNet
		Chainer
		DeepLearning4J
		Microsoft Cognitive Toolkit (CNTK)
	Distributed DL Processing using Elephas
	Alternate Framework for Distributed Deep Learning
		Distributed Keras
		TensorFlowOnSpark
		BigDL
		DeepLearning pipelines
		Zoo-analytics
	Deep Learning Operations (DLOps)
	Conclusion
10. Computer Vision with Apache Spark
	Introduction
	Structure
	Objectives
	Evolution of Computer Vision
	Defining an Image
	Different Formats of Image
	Annotation ways in CV
		Bounding Boxes (BB)
		3D cuboids
		Polygons-Based Annotation
		Lines and Splines
		Semantic Segmentation
		Key-Point and Landmark
		Circle
	Computer Vision Libraries
		Open-source Computer Vision Library (OpenCV)
		Imutils
		Scikit-Image
		Python-Tesseract (Pytessarct)
		PyTorchCV
		SimpleCV
		BoofCV
		IPSDK
		Python-Tesseract (Pytessarct)
	Components of Computer Vision
		Object Classification
		Object Detection
		Object Segmentation
		Object Tracking
	Convolution Neural Network (CNN) and its Working
		Convolution Operation
		Rectified Linear Unit (ReLu)
		Pooling
		Flattening
		Full Connection
		SoftMax and Cross-Entropy
	Timeline of the CNN Architecture
	Implementation of Distributed Processing in Image Classification using Google Colab
		Flow Chart of the codebase
		Output Snippet
	Real-time Computer Vision Pipeline
	Advancement in CV
		Generative Adversarial Network (GAN)
		Zero-Shot Learning (ZSL)
		Contrastive Learning (CL)
	Data Augmentation (DA) in CV
		Flipping
			Color Space
		Cropping
		Rotation
		Noise Injection
		Kernel Filters and Mixing Images (MI)
		Random Erasing
		Adversarial Training and GAN-based DA
		Neural Style Transfer (NST)
		Smart Augmentation (SA)
	Applications of CV
	Conclusion
Index