The Matrix Calculus You Need for Deep Learning

1 Introduction
2 Review: Scalar derivative rules
3 Introduction to vector calculus and partial derivatives
4 Matrix calculus
	4.1 Generalization of the Jacobian
	4.2 Derivatives of vector element-wise binary operators
	4.3 Derivatives involving scalar expansion
	4.4 Vector sum reduction
	4.5 The Chain Rules
		4.5.1 Single-variable chain rule
		4.5.2 Single-variable total-derivative chain rule
		4.5.3 Vector chain rule
5 The gradient of neuron activation
6 The gradient of the neural network loss function
	6.1 The gradient with respect to the weights
	6.2 The derivative with respect to the bias
7 Summary
8 Matrix Calculus Reference
	8.1 Gradients and Jacobians
	8.2 Element-wise operations on vectors
	8.3 Scalar expansion
	8.4 Vector reductions
	8.5 Chain rules
9 Notation
10 Resources