Multi-faceted Deep Learning: Models and Data

Preface
Acknowledgments
Contents
1 Introduction
2 Deep Neural Networks: Models and Methods
	2.1 Artificial Neural Networks
		2.1.1 Formal Neuron
			2.1.1.1 Activation Functions
		2.1.2 Artificial Neural Networks and Deep Neural Networks
	2.2 Convolutional Neural Networks
		2.2.1 General Principles
		2.2.2 Layers of a CNN
			2.2.2.1 Convolutional Layers
			2.2.2.2 Max-Pooling layers
			2.2.2.3 Dropout
		2.2.3 Some Well-Known CNNs Architectures
			2.2.3.1 LeNet Architecture and MNIST Dataset
			2.2.3.2 AlexNet Architecture
			2.2.3.3 GoogLeNet
			2.2.3.4 Other Important Architectures
	2.3 Optimization Methods
		2.3.1 Gradient Descent
		2.3.2 Stochastic Gradient Descent
		2.3.3 Momentum Based SGD
		2.3.4 Nesterov Accelerated Gradient Descent
		2.3.5 Adaptative Learning Rate
		2.3.6 Extensions of Gradient Descent
	2.4 Gradient Estimation in Neural Networks
	2.5 Recurrent Neural Networks
		2.5.1 General Principles
		2.5.2 Long-Short Term Memory Networks
	2.6 Generative Adversary Networks
	2.7 Autoencoders
	2.8 Siamese Neural Networks
	2.9 Conclusion
	References
3 Deep Learning for Semantic Segmentation
	3.1 Introduction
	3.2 Semantic Segmentation Challenges
	3.3 Traditional Approaches for Semantic Segmentation
	3.4 Semantic Segmentation Deep Learning Approaches
		3.4.1 Supervised Learning Approaches
			3.4.1.1 Transfer Learning Based Networks
			3.4.1.2 Learning Without Prior Knowledge
			3.4.1.3 Performance Metrics and Loss Functions
		3.4.2 Unsupervised and Weakly Supervised Learning
	3.5 Model Refinements
		3.5.1 Block Level Enhancement
		3.5.2 Attention Processes
		3.5.3 Multi Task Learning
			3.5.3.1 Semantic Segmentation as an Auxiliary Task
			3.5.3.2 Auxiliary Tasks for Improved Semantic Segmentation
	3.6 Data, Benchmarks and Model Evolution
		3.6.1 Importance of the Data and Public Collections
		3.6.2 A Way to Follow the State of the Art
		3.6.3 Typical Benchmarks
	3.7 Conclusion
	References
4 Beyond Full Supervision in Deep Learning
	4.1 Context
		4.1.1 Weakly Supervised Learning (WSL)
		4.1.2 Semi Supervised Learning (SSL)
		4.1.3 Self-training
	4.2 Negative Evidence Models for WSL
		4.2.1 Notations
		4.2.2 Negative Evidence Model
		4.2.3 Learning Formulation
		4.2.4 Negative Evidence Intuition
		4.2.5 ResNet-WELDON Network Architecture
			4.2.5.1 Feature Extraction Network
			4.2.5.2 Prediction Network Design
		4.2.6 Learning and Instantiations
		4.2.7 Experiments
	4.3 Beyond Reconstruction in Semi-supervised Learning
		4.3.1 Designing the HybridNet Architecture
		4.3.2 Training HybridNet
			4.3.2.1 Classification
			4.3.2.2 Reconstruction Losses
			4.3.2.3 Branch Cooperation
			4.3.2.4 Encouraging Invariance in the Discriminative Branch
		4.3.3 Experiments
	4.4 Medical Image Segmentation with Partial Labels
		4.4.1 Training from Partial Annotations with SMILE
		4.4.2 Self-supervision and Pseudo-Labeling
		4.4.3 Experiments
	4.5 Conclusion
	References
5 Similarity Metric Learning
	5.1 Introduction
	5.2 Metric Learning with Neural Networks
		5.2.1 Architectures
		5.2.2 Training Set Selection
			5.2.2.1 Pairs
			5.2.2.2 Triplets
			5.2.2.3 Tuples
		5.2.3 Loss Functions
			5.2.3.1 Cosine Pair-Wise
			5.2.3.2 Triangular
			5.2.3.3 Norm-Based
			5.2.3.4 Triplet
			5.2.3.5 Angular
			5.2.3.6 Deviance
			5.2.3.7 Quadruplets
			5.2.3.8 Tuples: Pair-Wise
			5.2.3.9 Tuples: Polar Sine
			5.2.3.10 SoftTriple
			5.2.3.11 Sphere Loss
			5.2.3.12 Probability-Driven
			5.2.3.13 Statistical
		5.2.4 Training Algorithms and Schemes
	5.3 Conclusion
	References
6 Zero-Shot Learning with Deep Neural Networks for Object Recognition
	6.1 Introduction
	6.2 Formalism, Settings and Evaluation
		6.2.1 Standard ZSL Setting
		6.2.2 Alternative ZSL Settings
		6.2.3 ZSL Evaluation
		6.2.4 Standard ZSL Datasets and Evaluation Biases
	6.3 Methods
		6.3.1 Ridge Regression Approaches
		6.3.2 Triplet-Loss Approaches
		6.3.3 Generative Approaches
	6.4 Semantic Features for Large Scale ZSL
	6.5 Conclusion and Current Challenges
	References
7 Image and Video Captioning Using Deep Architectures
	7.1 Introduction
	7.2 Basics of Visual Captioning
		7.2.1 From Neural Machine Translation to Visual Captioning
		7.2.2 Neural Language Models for Image and Video Captioning
		7.2.3 Building a Deep Caption Generation Model: A Generic Method
			7.2.3.1 Encoding Images and Videos
			7.2.3.2 Decoding Images and Videos
		7.2.4 Improving Captioning Models with Attention
	7.3 Optimization of Visual Captioning Models
		7.3.1 Pretraining Visual Features
		7.3.2 Optimizing the Language Model with a Cross-Entropy Loss
		7.3.3 Optimizing the Language Model by Reinforcement Learning
		7.3.4 Regularizing Captioning Models
			7.3.4.1 Matching Regularization
			7.3.4.2 Attribute Prediction
		7.3.5 Improving Captions at Inference Time
			7.3.5.1 Greedy Search vs Beam Search
			7.3.5.2 Captions Reranking
	7.4 Evaluation of Captions Quality
		7.4.1 BLEU-n
		7.4.2 ROUGEL
		7.4.3 METEOR
		7.4.4 CIDErD
	7.5 Captioning Datasets
		7.5.1 Image Captioning Datasets
		7.5.2 Video Captioning Datasets
	7.6 Results Reported in Published Works
		7.6.1 Image Captioning
		7.6.2 Video Captioning
	7.7 Other Related Works
		7.7.1 Image Dense Captioning
		7.7.2 Video Dense Captioning
		7.7.3 Movie Captioning
	7.8 Conclusion
	References
8 Deep Learning in Video Compression Algorithms
	8.1 Introduction
	8.2 Video Compression Standards
	8.3 Using Neural Networks for Video Compression
	8.4 Improving Intra and Inter Predictions Using Neural Networks
	8.5 Holistic Approaches
	8.6 Summary
	References
9 3D Convolutional Networks for Action Recognition: Application to Sport Gesture Recognition
	9.1 Introduction
	9.2 Highlights on Action Recognition Problem
		9.2.1 Action Classification from Videos with Handcrafted Features
		9.2.2 The Move to DNNs in Action Recognition Problem
		9.2.3 2D Convolutional Neural Networks for Action Classification
		9.2.4 From 2D to 3D ConvNets in Action Classification
		9.2.5 3D Convolutional Neural Networks for Action Classification
		9.2.6 Video Understanding for Racket Sports
	9.3 Datasets for Action Recognition
		9.3.1 Annotation Processes
			9.3.1.1 Automatic Annotation
			9.3.1.2 Manual Annotation
		9.3.2 Datasets for Action Classification
			9.3.2.1 The Acquisition-Controlled Datasets
			9.3.2.2 Datasets from Movies
			9.3.2.3 In-the-Wild Datasets
		9.3.3 The TTStroke-21 Dataset
			9.3.3.1 TTStroke-21 Acquisition
			9.3.3.2 TTStroke-21 Annotation
			9.3.3.3 Negative Samples Extraction
			9.3.3.4 Data for Evaluation
	9.4 TSTCNN: A Twin Spatio-Temporal 3D Convolutional Neural Network for Action Recognition
		9.4.1 Results
	9.5 Conclusion and Perspectives
	References
10 Deep Learning for Audio and Music
	10.1 Introduction
	10.2 DNN Architectures for Audio Processing
		10.2.1 DNN Architectures
		10.2.2 DNN Meta-architectures
		10.2.3 DNN Training Paradigms and Losses
	10.3 DNN Inputs for Audio Processing
		10.3.1 Using Time and Frequency Representations as Input
			10.3.1.1 Spectrogram Images Versus Natural Images
			10.3.1.2 DNN Models for Time and Frequency Representations as Inputs
		10.3.2 Using Waveform Representations as Input
		10.3.3 Using Knowledge-Driven Representations as Input
	10.4 Applications
		10.4.1 Music Content Description
		10.4.2 Environmental Sounds Description
		10.4.3 Content Processing: Source Separation
		10.4.4 Content Generation
		10.4.5 Semi-Supervised Learning and Self-Supervised Learning
			10.4.5.1 Semi-Supervised Learning
		10.4.6 Self-Supervised Learning
	10.5 Conclusion and Future Directions
	References
11 Explainable AI for Medical Imaging: Knowledge Matters
	11.1 Introduction
		11.1.1 A Matter of Trust
		11.1.2 The Emergence of XAI
		11.1.3 The Case of Medical Imaging
	11.2 The Augmented Pathologist
		11.2.1 Explainable Human Intelligence?
		11.2.2 Data and Model Visualization
		11.2.3 Safety and Robustness Improvement
	11.3 Investigating Alzheimer\'s Disease with CAM
	11.4 Breast Cancer Identification Using Deep Learning Approaches
		11.4.1 Breast Cancer Description
		11.4.2 Deep Learning and Breast Cancer Databases
			11.4.2.1 Deep Learning Architecture
			11.4.2.2 Breast Cancer Database
		11.4.3 Identification Results
	11.5 Conclusion
	References
12 Improving Video Quality with Generative Adversarial Networks
	12.1 Introduction
	12.2 Related Works
		12.2.1 Video and Image Restoration
		12.2.2 Video and Image Compression
		12.2.3 Hybrid Approaches
		12.2.4 Quality Metrics
	12.3 Generative Adversarial Networks vs Standard Enhancement CNNs
		12.3.1 Network Architectures
			12.3.1.1 Fully convolutional Generator
			12.3.1.2 Improving the Efficiency of Enhancement Architectures
			12.3.1.3 Discriminative Network
		12.3.2 Loss Functions
			12.3.2.1 Pixel-Wise MSE Loss
			12.3.2.2 SSIM Loss
			12.3.2.3 Perceptual Loss
			12.3.2.4 Adversarial Patch Loss
			12.3.2.5 Relativistic GAN
		12.3.3 Quality Agnostic Artifact Removal
		12.3.4 NoGAN Training
	12.4 Exploiting Transmitter and Receiver for Improvement
		12.4.1 Semantic Video Encoding
		12.4.2 Video Restoration
	12.5 Conclusion
	References
13 Conclusion