Library
Machine Learning

Deriving the Output Weight Update in Backpropagation

This lesson breaks down the gradient descent update for the output layer of a neural network by decomposing the error into local factors using the chain rule.

Loss Landscape
L=12∥y−a∥2L = \tfrac{1}{2} \|y - a\|^2
Imagine we have a neural network. We want to know how much changing a single weight in our output layer changes our overall prediction error.
Step-by-step solver
1

(a) Setup

Define the network layers: pre-activation z, sigmoid activation a, and the loss L.

z=W2h+b2,a=σ(z),L=12∑(yk−ak)2z=W_2h+b_2, \quad a=\sigma(z), \quad L=\tfrac{1}{2}\sum (y_k-a_k)^2
2

(b) Chain Rule

Expand the derivative of the loss with respect to output weights using the chain rule.

∂L∂W2=∂L∂a∂a∂z∂z∂W2\frac{\partial L}{\partial W_2} = \frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial W_2}
3

(c) Component derivatives

Calculate individual gradients: loss to output, output to pre-activation, and pre-activation to weights.

∂L∂a=(a−y),∂a∂z=a(1−a),∂z∂W2=hT\frac{\partial L}{\partial a} = (a-y), \quad \frac{\partial a}{\partial z} = a(1-a), \quad \frac{\partial z}{\partial W_2} = h^T
4

(d) Final Rule

Combine components into the delta term and form the weight update rule.

δ=(a−y)⊙a(1−a),W2←W2−ηδhT\delta = (a-y) \odot a(1-a), \quad W_2 \leftarrow W_2 - \eta \delta h^T

Original question

Derive the backpropagation weight update rule for the output layer of a 2-layer neural network with sigmoid activation and a mean squared error loss.

Follow-up chat

Ask me anything about this lesson. I'll answer using what we just covered.