Recurrent Neural Networks: From Simple to Gated Architectures

Preface
Contents
Notation
	Numbers and Arrays
	Indexing
	Linear Algebra Operations
	Sets and Graphs
	Calculus
	Probability and Information Theory
	Functions
	Datasets and Distributions
Introduction to the Book
Part I Basic Elements of Neural Networks
1 Network Architectures
	1.1 The Elements of Neural Models
		1.1.1 The Elements of a Biological Neuron
		1.1.2 Models of a Neuron—Simple Forms
		1.1.3 The Connection Parameters: Weights
		1.1.4 Activation Functions: Types of Nonlinearities
	1.2 Network Architectures: Feedforward, Recurrent, and Deep Networks
		1.2.1 Feedback Neural Networks: Vector Form
		1.2.2 An Alternate View Diagram of Feedforward Network Architectures
		1.2.3 Convolution Connection
			1.2.3.1 Convolutional Operations in 1D
			1.2.3.2 Tensor Processing
		1.2.4 Deep Learning or Deep Networks
2 Learning Processes
	2.1 Adaptive Learning
		2.1.1 Learning Types
	2.2 Optimization of Loss Functions
		2.2.1 The Stochastic Gradient Descent (SGD)
	2.3 Popular Loss Functions
	2.4 Appendix 2.1: Gradient System Basics
	2.5 Appendix 2.2: The LMS Algorithm
Part II Recurrent Neural Networks (RNN)
3 Recurrent Neural Networks (RNN)
	3.1 Simple Recurrent Neural Networks (sRNN)
		3.1.1 The (Stochastic) Gradient Descent for sRNN: The Backpropagation Through Time (BPTT))
	3.2 Basic Recurrent Neural Networks (bRNN)—the General Case
	3.3 Basic Recurrent Neural Networks (bRNN)—a Special Case
	3.4 Basic Recurrent Neural Networks (bRNN): Summary Equations
	3.5 Concluding Remarks
	3.6 Appendix 3.1: Global Stability of bRNN
	3.7 Appendix 3.2: Update Laws Derivation Details
Part III Gated Recurrent Neural Networks: The LSTM RNN
4 Gated RNN: The Long Short-Term Memory (LSTM) RNN
	4.1 Introduction and Background
	4.2 The Standard LSTM RNN
		4.2.1 The Long Short-Term Memory (LSTM) RNN
	4.3 Slim LSTMs: Reductions within the Gate(s)
		4.3.1 The Rationale in Developing the Slim LSTMs
		4.3.2 Variant 1: The LSTM_1 RNN
		4.3.3 Variant 2: The LSTM_2 RNN
		4.3.4 Variant 3: The LSTM_3 RNN
		4.3.5 Variant 4: The LSTM_4 RNN
		4.3.6 Variant 5: The LSTM_5 RNN
	4.4 Comparative Experiments of LSTM RNN Variants
		4.4.1 Experiments on the MNIST Dataset
		4.4.2 Experiments on the IMDB Dataset
	4.5 Concluding Remarks
Part IV Gated Recurrent Neural Networks: The GRU and The MGU RNN
5 Gated RNN: The Gated Recurrent Unit (GRU) RNN
	5.1 Introduction and Background
	5.2 The Standard GRU RNN
		5.2.1 The Long Short-Term Memory (LSTM) RNN
		5.2.2 The Gated Recurrent Unit (GRU) RNN
		5.2.3 The Gated Recurrent Unit (GRU) RNN vs. the LSTM RNN
	5.3 Slim GRU RNN: Reductions within the Gate(s)
		5.3.1 Variant 1: The GRU_1 RNN
		5.3.2 Variant 2: The GRU_2 RNN
		5.3.3 Variant 3: The GRU_3 RNN
		5.3.4 Variant 4: The GRU_4 RNN
		5.3.5 Variant 5: The GRU_5 RNN
	5.4 Sample Comparative Performance Evaluation
		5.4.1 Application to MNIST Dataset (Pixel-Wise)
		5.4.2 Application to MNIST Dataset (Row-Wise)
		5.4.3 Application to the IMDB Dataset (Text Sequence)
	5.5 Concluding Remarks
6 Gated RNN: The Minimal Gated Unit (MGU) RNN
	6.1 Introduction and Background
		6.1.1 Simple RNN Architectures
		6.1.2 LSTM RNN
		6.1.3 GRU RNN
		6.1.4 Gated Recurrent Unit (GRU) RNNs vs. the LSTM RNNs
	6.2 The Standard MGU RNN
	6.3 Slim MGU RNN: Reductions within the Gate(s)
		6.3.1 Variant 1: MGU_1 RNN
		6.3.2 Variant 2: MGU_2 RNN
		6.3.3 Variant 3: MGU_3 RNN
		6.3.4 Variant 4: MGU_4 RNN
		6.3.5 Variant 5: MGU_5 RNN
	6.4 Sample Comparative Performance Evaluation
		6.4.1 Sample Comparative MGU RNN Performance
		6.4.2 The Network Architecture
		6.4.3 Comparative Performance on the MNIST Dataset
		6.4.4 Reuters Newswire Topics Dataset
		6.4.5 Summary Discussion
	6.5 Concluding Remarks
References
Index