Backpropagation as Credit Assignment
How gradients move credit through a computation graph, where they fail, and what to check when training is unstable.
Backpropagation is reverse-mode automatic differentiation applied to a scalar loss. The useful mental model is bookkeeping over a computation graph: each local derivative tells one upstream operation how much it contributed to the final loss.
Computation graph model
A feedforward model composes functions:
\[f(x;\theta) = f_L(f_{L-1}(\dots f_1(x)))\]Training chooses parameters $\theta$ that reduce an empirical objective:
\[J(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell(f(x_i;\theta), y_i)\]Backpropagation computes $\nabla_\theta J$ by traversing the graph backward. For a chain $z = f(y)$ and $y = g(x)$:
\[\frac{\partial z}{\partial x} = \frac{\partial z}{\partial y}\frac{\partial y}{\partial x}\]For vector-valued intermediates, the local derivative is a Jacobian. Reverse mode avoids materializing most Jacobians. It propagates vector-Jacobian products from the scalar loss back to parameters.
For a layer:
\[a_l = W_l h_{l-1} + b_l, \qquad h_l = \phi(a_l)\]and upstream signal $\delta_l = \partial J / \partial a_l$, the common gradients are:
\[\frac{\partial J}{\partial W_l} = \delta_l h_{l-1}^\top, \qquad \frac{\partial J}{\partial b_l} = \delta_l\]The error signal to the previous layer is:
\[\delta_{l-1} = (W_l^\top \delta_l) \odot \phi'(a_{l-1})\]This equation explains most training pathologies. The gradient is repeatedly multiplied by weights and activation derivatives.
Failure modes
Vanishing gradients occur when repeated local factors have norms below one. Saturating sigmoid and tanh activations make this likely because their derivatives approach zero away from the origin.
Exploding gradients occur when repeated local factors have large norms. The symptom is not only a large loss. It can be NaNs, unstable optimizer moments, or a model that only trains with a tiny learning rate.
Nondifferentiable operations are not automatically wrong. ReLU has an undefined derivative at zero, but frameworks use a subgradient convention and train it successfully. Problems appear when a discrete choice blocks all useful signal, such as argmax inside the training path.
Broadcasting and reduction bugs are quiet. A loss reduced over the wrong axis can produce a scalar and a gradient that look valid while optimizing the wrong quantity.
Practical checks
Use gradient checks on custom operations. For a parameter direction $u$ with small $\epsilon$:
\[\frac{J(\theta + \epsilon u) - J(\theta - \epsilon u)}{2\epsilon} \approx u^\top \nabla_\theta J\]This finite-difference check is expensive and numerically delicate, so use it on small deterministic inputs, not in the training loop.
Log gradient norms per module:
\[\lVert g_l \rVert_2 = \sqrt{\sum_j g_{l,j}^2}\]A block with a zero norm is disconnected, saturated, frozen, or behind a dead branch. A block with a norm orders of magnitude above its neighbors is a candidate for clipping, normalization, initialization review, or learning-rate reduction.
Check loss scaling semantics. If the loss is averaged over a batch, doubling the batch size does not double the gradient. If the loss is summed, it does. This changes the effective learning rate.
Design guidance
Keep the differentiable path boring. Prefer smooth losses, stable normalizations, residual paths, and simple reductions. Move sampling, ranking, thresholding, and business decisions outside the gradient path unless the method explicitly handles them.
When training fails, inspect the graph before changing the model family. Verify tensor shapes, reduction axes, requires_grad flags, frozen parameters, gradient norms, and loss scale. A wrong graph can make a good architecture look bad.