Loss Functions and Optimization Behavior
What common losses reward, how gradients behave, and why metric alignment matters more than loss names.
A loss function defines the pressure applied during training. It is not the same thing as the final product metric. A good loss gives useful gradients and pushes the model toward decisions that matter under deployment costs.
Empirical risk
Most supervised training minimizes empirical risk:
\[\hat{R}(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell(f_\theta(x_i), y_i)\]Regularized training adds a penalty:
\[J(\theta) = \hat{R}(\theta) + \lambda \Omega(\theta)\]For $L_2$ regularization, $\Omega(\theta)=\lVert\theta\rVert_2^2$. This changes the optimizer’s preference among solutions that fit the training data.
Regression losses
Mean squared error:
\[\text{MSE} = \frac{1}{N}\sum_i (y_i - \hat{y}_i)^2\]MSE heavily penalizes large residuals. It is a natural fit when the conditional mean is the desired target and large errors should dominate the objective.
Mean absolute error:
\[\text{MAE} = \frac{1}{N}\sum_i |y_i - \hat{y}_i|\]MAE targets the conditional median and is less sensitive to outliers. Its gradient is less smooth near zero, which can matter for optimization.
Huber loss uses quadratic behavior near zero and linear behavior outside a threshold $\delta$:
\[L_\delta(r) = \begin{cases} \frac{1}{2}r^2 & |r| \le \delta \\ \delta(|r| - \frac{1}{2}\delta) & |r| > \delta \end{cases}\]Use it when small residual precision matters but outliers should not dominate every update.
Classification losses
For binary classification with probability $p$ assigned to class $1$:
\[\ell(p,y) = -y\log(p) - (1-y)\log(1-p)\]For multiclass classification with logits $z$ and softmax probabilities:
\[p_k = \frac{e^{z_k}}{\sum_j e^{z_j}}, \qquad \ell = -\log p_y\]Cross entropy rewards probability assigned to the correct label, not just the final class decision. For logits plus softmax cross entropy, the gradient has the useful form:
\[\frac{\partial \ell}{\partial z_k} = p_k - \mathbb{1}[k=y]\]A numerically stable softmax subtracts the maximum logit:
\[\text{softmax}(z)_k = \frac{e^{z_k - m}}{\sum_j e^{z_j - m}}, \qquad m = \max_j z_j\]This preserves probabilities and avoids overflow.
Metric alignment
A model can improve cross entropy while the product decision gets worse. The loss optimizes prediction quality. The application often cares about a thresholded decision under asymmetric costs.
For a binary decision with false-positive cost $C_{FP}$ and false-negative cost $C_{FN}$, the threshold for predicting positive from a calibrated probability is:
\[t = \frac{C_{FP}}{C_{FP} + C_{FN}}\]This threshold assumes calibrated probabilities and equal utility outside the two error costs. If the model’s probabilities are not calibrated, moving the threshold may improve the validation metric but fail under distribution shift.
Practical checks
- Compare loss curves against the decision metric. If they diverge, write down which one owns the launch decision.
- Inspect per-class or per-slice losses. A low average can hide a failed minority class.
- Check logits for overflow, saturation, and extreme margins.
- Report a baseline. A sophisticated model that barely beats a constant predictor is not evidence of a useful signal.
- Keep reduction semantics explicit: mean over examples, tokens, pixels, or all elements.
Design guidance
Choose the loss for gradient quality and the metric for decision quality. They should be related, but they are not interchangeable. When the deployment decision has asymmetric cost, tune and report the decision threshold separately from the training loss.