Train Loss vs Eval Loss: Reading the Gap
How to tell healthy training from overfitting by comparing train loss and eval loss, with concrete gap thresholds and the one signal that actually matters.
Quick Navigation
Difficulty: Beginner
Estimated Time: 10-15 minutes
Prerequisites: Basic machine learning concepts, Familiarity with model training, Understanding of loss functions
Two numbers scroll past during every training run, and the relationship between them tells you almost everything about whether the run is going well.
What They Are
Train loss measures how wrong the model is on data it is currently learning from. It is typically logged every few steps, so it is noisy but immediate.
Eval loss measures how wrong the model is on held-out data it has never seen. It is computed less often, for example every 500 steps against a fixed evaluation set, because it costs a full pass over that set.
The analogy that sticks: train loss is the homework score, eval loss is the exam score. A student who has memorised the answer key aces the homework and fails the exam.
The Rule
Train loss should be equal to or slightly below eval loss. The model always performs a little better on data it trained on, and that is normal, not a warning sign.
What matters is the size of the gap.
Overfitting Thresholds
| Gap | Meaning | Example | Action |
|---|---|---|---|
| Under 0.05 | No overfitting | train=0.40, eval=0.41 | Keep going |
| 0.05 to 0.15 | Mild overfitting | train=0.30, eval=0.42 | Watch closely |
| 0.15 to 0.30 | Overfitting | train=0.20, eval=0.45 | Consider stopping |
| Over 0.30 | Severe overfitting | train=0.10, eval=0.50 | Stop, use an earlier checkpoint |
Treat these as rules of thumb rather than hard constants. The absolute loss scale differs between tasks and loss functions, so a 0.05 gap means something different at a base loss of 0.4 than at a base loss of 4.0.
The Key Signal: Is Eval Loss Going Up?
The gap alone can mislead. The direction of eval loss is the reliable signal.
- Not overfitting — eval loss is flat or still decreasing, even if the gap grows slightly. The model is still learning something that transfers.
- Overfitting — eval loss is increasing while train loss decreases. The model is memorising rather than learning, and every additional step makes it worse on real data.
That second pattern is the one worth setting an alert on. A widening gap where eval loss still falls is often fine; a narrowing gap where eval loss rises is not.
A Worked Example
Consider a single-epoch fine-tune of a large instruction model on a synthetic instruction dataset:
| Step | Train loss | Eval loss | Gap |
|---|---|---|---|
| 500 | 0.410 | 0.412 | 0.002 |
A gap of 0.002 is essentially zero. The model generalises to held-out data as well as it fits the training data. With only one epoch over the corpus, the model never sees an example twice, so there is very little opportunity to memorise in the first place. Overfitting risk here is minimal, and the run should continue.
This is a common property of single-epoch training on large corpora: the gap stays near zero and overfitting simply is not the failure mode you need to watch for. Multi-epoch fine-tuning on a small dataset is the opposite case, and that is where these thresholds earn their keep.
Quick Reference
| Situation | Meaning |
|---|---|
| train slightly below eval | Healthy. The model generalises well. |
| train roughly equal to eval | Ideal. No overfitting. |
| train far below eval | Overfitting. The model has memorised. |
| train above eval | Unusual. Acceptable if eval loss is low, otherwise check for a data split or regularisation quirk. |
That last row surprises people. Train loss above eval loss usually means dropout or other regularisation is active during training but not evaluation, or that the evaluation set happens to be easier than the training distribution. It is worth understanding, but it is not overfitting.
What to Do When the Gap Grows
- Stop early and keep an earlier checkpoint. The cheapest fix, and the reason to checkpoint frequently rather than only at the end.
- Reduce epochs before reaching for anything more elaborate. Most small-dataset overfitting is simply too many passes.
- Add data if you can. More unique examples beat almost every regularisation technique.
- Increase regularisation with dropout, weight decay, or a lower learning rate.
- Check the split first. A gap that appears immediately, at step one, is usually a distribution mismatch between train and eval sets, not overfitting at all.
Tags: #MachineLearning #Training #Overfitting #ModelEvaluation