Deep Learning and Parallel Computing Environment for Bioengineering Systems

Cover
Deep Learning and
Parallel Computing
Environment for
Bioengineering
Systems
Copyright
List of Contributors
Preface
	Organization of the Book
	Audience
Foreword
Acknowledgment
Contents
1 Parallel Computing, Graphics Processing Unit (GPU) and New Hardware for Deep Learning in Computational Intelligence Research
	1.1 Introduction
		1.1.1 Machine and Deep Learning
		1.1.2 Graphics Processing Unit (GPU)
		1.1.3 Computational Intelligence
		1.1.4 GPU, Deep Learning and Computational Intelligence
	1.2 Deep Learning and Parallelization
		1.2.1 Parallel Processing Concepts
		1.2.2 Deep Learning Using Parallel Algorithms
			1.2.2.1 Understanding the Needs and Benefits of Parallel Algorithms in Deep Learning
			1.2.2.2 Challenges in Implementing Parallel Algorithms in Deep Learning
		1.2.3 Parallelization Methods to Distribute Computation Across Multiple Machines
			1.2.3.1 Local Training
			1.2.3.2 Distributed Training
		1.2.4 Methods to Train Deep Neural Networks Using Parallelization
			1.2.4.1 Inter-Model Parallelism
			1.2.4.2 Data Parallelism
			1.2.4.3 Intra-Model Parallelism
		1.2.5 Parallelization Over Data, Function, Parameter and Prediction Scale
			1.2.5.1 Data Partitioning
			1.2.5.2 Function Partitioning
			1.2.5.3 Hyperparameter Learning
			1.2.5.4 Prediction at Scale
		1.2.6 Types of Speed-Up and Scaling
			1.2.6.1 Problem Constrained (PC) Scaling
			1.2.6.2 Time Constrained (TC) Scaling
	1.3 Role of Graphics Processing Unit in Parallel Deep Learning
		1.3.1 Hardware Architecture of CPU and GPU
			1.3.1.1 Conventional CPU Architecture
			1.3.1.2 Modern GPU Architecture
		1.3.2 Suitability of GPU to Parallel Deep Learning
		1.3.3 CPU vs. GPU
		1.3.4 Advantages of Using GPU in Parallel Deep Learning
		1.3.5 Disadvantages of Using GPU in Parallel Deep Learning
		1.3.6 Famous GPUs on the Market
			1.3.6.1 NVIDIA
			1.3.6.2 AMD
	1.4 GPU Based Parallel Deep Learning on Computational Intelligence Applications With Case Study
		1.4.1 Dataflow of the Deep Parallelized Training and Testing of Computational Intelligence Applications
		1.4.2 Numerical Example for a Generic Computational Intelligence Application
		1.4.3 Dealing With Limited GPU Memory
		1.4.4 Computational Intelligence Applications
	1.5 Implementation Screenshots to Visualize the Training Process
	1.6 Summary
	References
2 Big Data Analytics and Deep Learning in Bioinformatics With Hadoop
	2.1 Introduction
	2.2 From Big Data to Knowledge Discovery With Hadoop
		2.2.1 Hadoop Big Data Framework
		2.2.2 Big Data Collection and Ingestion
			2.2.2.1 Apache Sqoop (SQL-to-Hadoop)
			2.2.2.2 Apache Flume
			2.2.2.3 Apache Kafka
		2.2.3 Data Staging and Storage on Hadoop
		2.2.4 Data Processing and Analysis Frameworks
			2.2.4.1 Batch Processing Only Framework - MapReduce
			2.2.4.2 Stream Processing Only Framework
			2.2.4.3 Hybrid Processing Framework
		2.2.5 Big Data Analysis and Visualization
	2.3 Machine Learning for Big Data Analysis
		2.3.1 Machine Learning Methods
			2.3.1.1 Supervised Machine Learning
			2.3.1.2 Unsupervised Machine Learning
			2.3.1.3 Semi-Supervised Machine Learning
			2.3.1.4 Reinforcement Learning
		2.3.2 Deep Learning and Neural Networks
			2.3.2.1 Artificial Neural Networks (ANNs)
		2.3.3 Machine Learning and Hadoop
			2.3.3.1 Spark, Mahout and MLlib
		2.3.4 Distributed Deep Learning and Hadoop
	2.4 Conclusions
	References
3 Image Fusion Through Deep Convolutional Neural Network
	3.1 Introduction
	3.2 Image Fusion
	3.3 Registration
		3.3.1 Image Registration Stages
			3.3.1.1 Feature Detection
			3.3.1.2 Feature Matching
			3.3.1.3 Transform Model Estimation
			3.3.1.4 Resampling
		3.3.2 Need for Image Registration in Medical Imaging
			3.3.2.1 SURF Based Registration
			3.3.2.2 BRISK Based Registration
			3.3.2.3 Implementation
	3.4 Existing Image Fusion Methods - Overview
	3.5 Deep Learning
	3.6 Convolutional Neural Network (CNN)
	3.7 CNN Based Image Fusion Algorithms
		3.7.1 Deep Stacked CNN (DSCNN)
		3.7.2 CNN Based Similarity Learning
		3.7.3 CNN Based Fusion Using Pyramidal Decomposition
	3.8 Evaluation Metrics
		3.8.1 Entropy (S)
		3.8.2 Standard Deviation (σ)
		3.8.3 Spatial Frequency (SF)
		3.8.4 Mutual Information (MI)
		3.8.5 Image Quality Index (IQI)
		3.8.6 QRS/F
		3.8.7 QG
		3.8.8 Structural Similarity Index Metric (SSIM)
		3.8.9 FSIM
		3.8.10 Contrast (C)
		3.8.11 QE
		3.8.12 Average Gradient (AG)
		3.8.13 Human Perception-Based Metric (QCB)
		3.8.14 Processing Time (T)
	3.9 Results Interpretation and Discussion
	3.10 Issues in Existing CNN Based Image Fusion Methods
	3.11 Conclusions
	References
4 Medical Imaging With Intelligent Systems: A Review
	4.1 Introduction
	4.2 Tumor Types and Grading
		4.2.1 Why Are Studies Concentrated Mostly on Gliomas?
		4.2.2 Grading
		4.2.3 Symptoms of a Brain Tumor
		4.2.4 Diagnosis and Treatment of a Brain Tumor
	4.3 Imaging Techniques
		4.3.1 Reading an MR Image
		4.3.2 Advanced Magnetic Resonance Imaging
			4.3.2.1 MRS
			4.3.2.2 Ultra-High-Field 7 T MRI
	4.4 Machine Learning (ML) - Supervised and Unsupervised Methods
		4.4.1 ML Software Packages/Toolboxes
	4.5 Deep Learning (DL)
		4.5.1 DL Tools/Libraries
	4.6 Evaluation and Validation Metrics
	4.7 Embedding Into Clinics
	4.8 Current State-of-the-Art
		4.8.1 Deep Learning Concepts for Brain Tumor Grading
	4.9 Discussion
	4.10 Conclusions
	Short Authors Biographies
	Acknowledgments
	Funding
	References
5 Medical Image Analysis With Deep Neural Networks
	5.1 Introduction
	5.2 Convolutional Neural Networks
	5.3 Convolutional Neural Network Methods
	5.4 Convolutional Layer
		5.4.1 Tiled Convolution
		5.4.2 Transposed Convolution
		5.4.3 Dilated Convolution
		5.4.4 Network-in-Network
		5.4.5 Inception Module
	5.5 Pooling Layer
		5.5.1 Lp Pooling
		5.5.2 Mixed Pooling
		5.5.3 Stochastic Pooling
		5.5.4 Spectral Pooling
		5.5.5 Spatial Pyramid Pooling
		5.5.6 Multi-Scale Orderless Pooling
	5.6 Activation Function
		5.6.1 Rectified Linear Unit (ReLU)
		5.6.2 Leaky ReLU
		5.6.3 Parametric ReLU
		5.6.4 Randomized ReLU
		5.6.5 Exponential Linear Unit (ELU)
		5.6.6 Maxout
		5.6.7 Probout
	5.7 Applications of CNN in Medical Image Analysis
		5.7.1 Image Classification
		5.7.2 Object Classification
		5.7.3 Region, Organ, and Landmark Localization
		5.7.4 Object or Lesion Detection
		5.7.5 Organ and Substructure Segmentation
		5.7.6 Lesion Segmentation
	5.8 Discussion
	5.9 Conclusions
	References
6 Deep Convolutional Neural Network for Image Classification on CUDA Platform
	6.1 Introduction
	6.2 Image Classification
		6.2.1 Image Classification Approach
		6.2.2 Image Classification Techniques
		6.2.3 Research Gaps
		6.2.4 Research Challenge
		6.2.5 Problem Definition
		6.2.6 Objective
	6.3 Deep Convolutional Neuron Network
	6.4 Compute Unified Device Architecture (CUDA)
	6.5 TensorFlow
	6.6 Implementation
		6.6.1 Deep Convolutional Neural Networks
		6.6.2 Dataset
		6.6.3 Implementing an Image Classifier
		6.6.4 Installation and System Requirements
		6.6.5 Algorithms
	6.7 Result Analysis
		6.7.1 Neural Networks in TensorFlow
		6.7.2 Understanding the Original Image Dataset
		6.7.3 Understanding the Original Labels
		6.7.4 Implementing Pre-process Functions
		6.7.5 Output of the Model
		6.7.6 Training a Model Using Multiple GPU Cards/CUDA
	6.8 Conclusions
	References
7 Efficient Deep Learning Approaches for Health Informatics
	7.1 Introduction
		Machine Learning Vs. Deep Learning
	7.2 Deep Learning Approaches
		Deep Autoencoders
		Recurrent Neural Networks (RNNs)
		Restricted Boltzmann Machine (RBM)
		Deep Belief Network
		Deep Boltzmann Machine (DBM)
		Convolutional Neural Network
	7.3 Applications
		Translational Bioinformatics
		Clinical Imaging
		Electronic Health Records
		Genomics
		Mobile Devices
	7.4 Challenges and Limitations
	7.5 Conclusions
	References
8 Deep Learning and Semi-Supervised and Transfer Learning Algorithms for Medical Imaging
	8.1 Introduction
	8.2 Image Acquisition in the Medical Field
	8.3 Deep Learning Over Machine Learning
	8.4 Neural Network Architecture
	8.5 Defining Deep Learning
	8.6 Deep Learning Architecture
		8.6.1 Convolution Neural Network (CNN)
		8.6.2 Recurrent Neural Network
		8.6.3 Deep Neural Network
	8.7 Deep Learning in Medical Imaging 8.7
		8.7.1 Diabetic Retinopathy
		8.7.2 Cardiac Imaging
		8.7.3 Tumor Classification in Homogeneous Breast Tissue
	8.8 Developments in Deep Learning Methods
		8.8.1 Black Box and Deep Learning
		8.8.2 Semi-Supervised and Transfer Learning Algorithms
			8.8.2.1 Semi-Supervised Learning
			8.8.2.2 Supervised Learning
			8.8.2.3 Unsupervised Learning
		8.8.3 Applications of Semi-Supervised Learning in Medical Imaging
			Pelvic MR Image Segmentation Based on Multi-Task Residual Fully Convolutional Networks
		8.8.4 Method
	8.9 Transfer Learning
		8.9.1 Transfer Learning in Image Data
			8.9.1.1 Mechanisms of Deep Transfer Learning for Medical Imaging
			8.9.1.2 Dataset and Training
			8.9.1.3 Transferred Learned Features
			8.9.1.4 Traditional Texture Feature Results
		8.9.2 Transfer Learning Technique for the Detection of Breast Cancer
			8.9.2.1 Dataset
			8.9.2.2 Preprocessing
			8.9.2.3 Transfer Learning Part
			8.9.2.4 Results
	References
9 Survey on Evaluating the Performance of Machine Learning Algorithms: Past Contributions and Future Roadmap
	9.1 Introduction
	9.2 Methodology
	9.3 Linear Regression
	9.4 Nonlinear Regression
		9.4.1 Support Vector Machine
		9.4.2 K-Nearest Neighbors
		9.4.3 Neural Network
	9.5 Nonlinear Decision Tree Regression
		9.5.1 Regression With Decision Trees
		9.5.2 Random Forest
	9.6 Linear Classification in R
	9.7 Results and Discussion
	9.8 Conclusions
	References
10 Miracle of Deep Learning Using IoT
	10.1 Introduction
	10.2 Inspiration
	10.3 Decisions in an Area of Deep Learning and IoT
		10.3.1 IoT Reference Model
		10.3.2 Rudiments of Machine Learning
		10.3.3 Algorithms for Efficient Training
		10.3.4 Secure Deep Learning
		10.3.5 Robust and Resolution-Invariant Image Classification
		10.3.6 Planning With Flawless and Noisy Images
		10.3.7 Smart and Fast Data Processing
	10.4 Simulation Results and Performance Analysis of Handwritten Digits Recognition in IoT
	10.5 An Intelligent Traffic Load Prediction
	10.6 Performance of Deep Learning Based Channel Assignment
		10.6.1 A Deep Learning System for the Individual Pursuit
		10.6.2 Network Architecture
		10.6.3 Dataset
		10.6.4 Statistics
		10.6.5 Effectiveness of Online Instance Matching
	10.7 Discussion
	10.8 Conclusion
	References
11 Challenges in Storing and Processing Big Data Using Hadoop and Spark
	11.1 Introduction
	11.2 Background and Main Focus
		11.2.1 Challenges of Big Data Technologies
			11.2.1.1 Memory Issues
			11.2.1.2 Limitations of Apache Big Data Technologies
		11.2.2 Real Time Applications of Big Data Frameworks
	11.3 Hadoop Architecture
	11.4 MapReduce Architecture
	11.5 Joins in MapReduce
	11.6 Apache Storm
	11.7 Apache Spark Environment
		11.7.1 Use Cases of Big Data Technologies
	11.8 Graph Analysis With GraphX
	11.9 Streaming Data Analytics
	11.10 Futer Research Directions
	11.11 Conclusion
	References
12 An Efficient Biography-Based Optimization Algorithm to Solve the Location Routing Problem With Intermediate Depots for Multiple Perishable Products
	12.1 Introduction
	12.2 Model Development
		12.2.1 Mathematical Model
			12.2.1.1 Linearization of the Nonlinear Equations
		12.2.2 An Illustration
	12.3 Solution Method
		12.3.1 Introduction to Biography Based Optimization Algorithm
			12.3.1.1 Migration Operator
			12.3.1.2 Mutation Operator
		12.3.2 Solution Representation
		12.3.3 Initial Solution Generation
		12.3.4 Immigration Phase
		12.3.5 Mutation Phase
		12.3.6 Optimal Parameter Design
	12.4 Computational Results
		12.4.1 Optimal Solutions for Instance Problems
		12.4.2 Sensitivity Analysis
	12.5 Discussion, Concluding Remarks and Future Research Directions
	References
13 Evolutionary Mapping Techniques for Systolic Computing System
	13.1 Introduction
	13.2 Systolic Arrays
	13.3 Evolutionary Algorithms
	13.4 Swarm Intelligence (SI)
	13.5 Mapping Techniques
	13.6 Systolic Implementation of Texture Analysis
	13.7 Results and Discussion
		13.7.1 Performance of EA for F8 Optimization
		13.7.2 Texture Analysis
	13.8 Conclusions
	List of Acronyms and Abbreviations
	References
14 Varied Expression Analysis of Children With ASD Using Multimodal Deep Learning Technique
	14.1 Introduction
	14.2 State-of-the-Art
	14.3 Methodology
		14.3.1 Detection of Human Faces
			14.3.1.1 Selection of Haar Features in Selected Faces
			14.3.1.2 Construction of an Integral Image
			14.3.1.3 AdaBoost Technique to Build Strong Classifiers
			14.3.1.4 Cascade Classifier
		14.3.2 Extraction of Features
		14.3.3 Expression Classifier
		14.3.4 Expression Identification Through a Convolution Neural Network (CNN)
			14.3.4.1 Convolution Layer
			14.3.4.2 Max Pool Layer
			14.3.4.3 Fully Connected Layer
	14.4 Results and Analysis
		14.4.1 Accuracy of the Face Expression Analysis
	14.5 Conclusions
	14.6 Future Work
	References
15 Parallel Machine Learning and Deep Learning Approaches for Bioinformatics
	15.1 Introduction
		15.1.1 Machine Learning and Deep Learning
		15.1.2 Role of Parallelization in Deep Learning
		15.1.3 Deep Learning Applications in Bioinformatics
	15.2 Deep Learning and Parallel Processing
		15.2.1 Parallel Processing
		15.2.2 Scalability of Parallelization Methods
		15.2.3 Deep Learning Using Parallel Algorithms
			15.2.3.1 Advantages of Using Parallelism in Deep Learning
			15.2.3.2 Parallelism Challenges in Deep Learning
	15.3 Deep Learning and Bioinformatics
		15.3.1 Bioinformatics Applications
			15.3.1.1 Sequence Analysis
			15.3.1.2 Genome Annotation
			15.3.1.3 Analysis of Gene Expression
			15.3.1.4 Analysis of Protein Expression
			15.3.1.5 Analysis of Mutations in Cancer
			15.3.1.6 Protein Structure Prediction
			15.3.1.7 Modeling Biological Systems
			15.3.1.8 High-Throughput Image Analysis
			15.3.1.9 Microarrays
			15.3.1.10 Systems Biology
			15.3.1.11 Text Mining
		15.3.2 Advantages of Using Parallel Deep Learning in Bioinformatics Applications
		15.3.3 Challenges in Using Parallel Deep Learning for Bioinformatics Applications
	15.4 Parallel Deep Learning in Bioinformatics Applications With Implementation and Real Time Numerical Example
	15.5 Sample Implementation Screenshots to Visualize the Training Process
	15.6 Summary
	References
Index
Back Cover