Abstract
Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses like cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we provide theoretical and empirical evidence that these two errors are not minimized simultaneously during training. Selecting the best training epoch based on validation loss thus leads to a compromise point that is suboptimal for both calibration error and, most importantly, refinement error. To address this, we introduce a new metric for early stopping and hyperparameter tuning that makes it possible to minimize refinement error during training. The calibration error is minimized after training, using standard techniques. Our method integrates seamlessly with any architecture and consistently improves performance across diverse classification tasks.
The Conflict in Standard Early Stopping
In machine learning classification, we aim for probabilities that are both informative and honest. These are measured by "proper scoring rules" like cross-entropy, which decompose into Refinement Error (class separation) and Calibration Error (confidence alignment).
A critical oversight in modern training is the assumption that these errors are minimized simultaneously. Empirical evidence shows that as a model trains, it often reaches its peak ability to separate classes (best refinement) long after its confidence scores have started to drift or overfit (poor calibration). Standard early stopping, which terminates training at the lowest total validation loss, forces a compromise between these two conflicting goals.
As seen in Figure 1, the calibration error starts rising while the refinement error is still decreasing. By stopping at the green dot (minimum validation loss), we leave significant refinement performance on the table. This "two minimizers" phenomenon suggests that standard early stopping kills training too early, sacrificing the model's ultimate predictive potential.
The "Refine, then Calibrate" Paradigm
To address this, we propose a new training paradigm: Refine, then Calibrate. Since calibration can be easily corrected post-hoc using simple techniques like Temperature Scaling (TS), training should focus exclusively on the "hard" task of refinement. Refinement represents the "intrinsic" power of the classifier—its potential performance once perfectly calibrated.
TS-Refinement Estimator
We propose estimating refinement error during training by applying a fast Temperature Scaling fit to the validation set at every epoch. Instead of stopping when raw validation loss is lowest, training continues until the validation loss after temperature scaling is minimized. This ensures the model is captured at its peak "expertise," regardless of its raw output calibration.
This approach is supported by high-dimensional asymptotics, proving that the optimal regularization for refinement differs from that for calibration. By decoupling these objectives, we can push models further into the training process to maximize their discriminative power.
Experimental Results
The approach was validated across diverse modalities:
- Computer Vision: Using ResNet-18 and WideResNet on CIFAR-10, CIFAR-100, and SVHN, refinement-based stopping consistently yielded lower test-time logloss compared to standard methods.
- Tabular Data: On a benchmark of 196 datasets, "Refine, then Calibrate" improved performance for XGBoost and neural networks (MLPs, RealMLP), particularly on larger datasets with over 10,000 samples.
- Efficiency: To make this practical, we developed a bisection-based implementation of Temperature Scaling in the
probmetricspackage. This method is significantly faster and more robust than standard L-BFGS optimization used in most libraries.
Theoretical models suggest that refinement-based stopping can reduce downstream cross-entropy by up to 25% in specific high-dimensional regimes where the two error terms diverge significantly.
Conclusion and Future Work
By decoupling refinement and calibration during the training process, we can achieve superior model performance. While highly effective, the refinement estimator can be sensitive to small validation sets, where it may overfit. Future research will explore cross-validation for more stable refinement estimates and investigate the implications for "long-course" training in Large Language Models. This work opens new avenues for optimizing learning rate schedulers and weight decay to specifically target refinement.
Summary updated