Model Evaluation Beyond Accuracy
Metric selection, confusion-matrix rates, calibration, uncertainty, and why accuracy is often the wrong launch gate.
Accuracy answers one narrow question: what fraction of predictions matched the label at one threshold. That is often insufficient. Evaluation should match the decision being made, the costs of being wrong, and the data slices where failure matters.
Confusion matrix rates
For binary classification:
| Term | Meaning |
|---|---|
| true positive | predicted positive and label positive |
| false positive | predicted positive and label negative |
| true negative | predicted negative and label negative |
| false negative | predicted negative and label positive |
Useful rates:
\[\text{precision} = \frac{TP}{TP + FP}\] \[\text{recall} = \frac{TP}{TP + FN}\] \[F_\beta = (1+\beta^2)\frac{\text{precision}\cdot\text{recall}}{\beta^2\text{precision}+\text{recall}}\]Precision measures how trustworthy positive predictions are. Recall measures how many true positives are found. $F_1$ is useful only when precision and recall deserve roughly symmetric treatment. It can hide the actual operating point.
Thresholds and ranking
Many classifiers output scores. A threshold turns a score into a decision:
\[\hat{y} = \mathbb{1}[s(x) \ge t]\]Changing $t$ changes precision and recall without retraining the model. ROC curves plot true-positive rate against false-positive rate across thresholds. Precision-recall curves are often more informative when positives are rare because precision exposes false positives directly.
Average precision summarizes a precision-recall curve as a weighted sum over recall changes:
\[AP = \sum_n (R_n - R_{n-1})P_n\]Use ranking metrics when the system shows several candidates and success means the correct one appears in the set.
Calibration
A calibrated model’s predicted probability matches observed frequency:
\[P(Y=1 \mid \hat{p}=p) = p\]If a model assigns 0.8 probability to 1000 similar cases, roughly 800 should be positive. Calibration matters for risk scoring, triage, thresholding, and combining model outputs with other decision logic.
Brier score is squared error on probabilities:
\[\text{Brier} = \frac{1}{N}\sum_i (\hat{p}_i - y_i)^2\]Log loss is stricter about confident wrong predictions:
\[\text{log loss} = -\frac{1}{N}\sum_i y_i\log \hat{p}_i + (1-y_i)\log(1-\hat{p}_i)\]Uncertainty and slices
Report confidence intervals or repeated-run variability when the evaluation set is small. For a simple binomial accuracy estimate:
\[SE \approx \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\]This approximation does not replace a careful study design, but it prevents over-reading tiny differences on small test sets.
Always slice metrics by meaningful subgroups: class, source, time, device, geography, data quality, prompt type, or traffic class. The average metric is not enough when one slice owns most of the risk.
Practical checks
- Compare against trivial baselines and the current production behavior.
- Choose the metric before model selection.
- Keep threshold tuning on validation data, not test data.
- Report confusion-matrix counts, not only derived rates.
- Inspect calibration when probabilities drive decisions.
- Track per-slice metrics and minimum acceptable slice performance.
Design guidance
Start from the decision. If the system must catch rare failures, optimize recall at a tolerable false-positive budget. If every false positive pages a human, precision and calibration may dominate. Accuracy is acceptable only when classes and costs are balanced enough for it to represent the real objective.