Train, Validation, and Test Leakage

Leakage means evaluation data influenced training, model selection, preprocessing, labeling, or feature construction. It produces numbers that look scientific and fail in deployment. The core rule: a prediction-time system may use only information available at prediction time.

Split roles

Use each split for one job:

A test set reused for repeated model selection becomes another validation set. Its metric is no longer an unbiased final estimate.

Preprocessing leakage

Fitting a transform on all data leaks test distribution into training. For a standard scaler:

$\mu$ and $\sigma$ must be estimated from the training split only. The same fitted transform is then applied to validation and test data.

The same rule applies to imputation, PCA, feature selection, vocabulary building, target encoding, outlier filtering, and learned normalization.

Duplicate and group leakage

Random row splits fail when rows are correlated. Examples:

• several crops from the same image,
• multiple windows from one time series,
• packets from the same flow,
• repeated measurements from one device,
• near-duplicate documents,
• user history split across train and test.

The split unit must match the independence assumption. If deployment predicts on unseen users, split by user. If deployment predicts future events, split by time.

Temporal leakage

A feature leaks when it uses future information. Rolling statistics are common offenders. A feature computed at time $t$ must depend only on observations at or before $t$:

If the feature uses $\tau > t$, the model is training with time travel.

Label windows create a second boundary. If the label looks ahead seven days, then examples near the split boundary can overlap unless the split includes an embargo window.

Benchmark overfitting

Public leaderboards leak through iteration. Every submission gives information about the test distribution. The more times a team adapts to that signal, the less the leaderboard measures generalization.

For internal work, the same failure happens when a team repeatedly tries variants and reports the best test result. Keep a locked final test or use nested validation when model selection is heavy.

Practical checks

• Split before fitting transforms.
• Use pipeline APIs that bind fit to the training fold during cross-validation.
• Split by group when examples share identity, source, or collection process.
• Split by time when deployment is time-forward.
• Deduplicate across splits using hashes or embedding similarity for text and images.
• Audit each feature with this question: would this value be known at prediction time?
• Keep a change log of every decision informed by validation or test metrics.

Design guidance

Leakage is an experimental-design bug, not a modeling bug. Fix the data boundary before tuning architecture. If the clean split drops performance sharply, believe the clean split.

Source anchors

• scikit-learn common pitfalls, data leakage
• Google Rules of Machine Learning, training-serving skew
• Kaufman et al., Leakage in Data Mining