Abstract
Modern neural networks are frequently overconfident in their predictions, necessitating post-hoc calibration methods like conformal inference to provide rigorous uncertainty guarantees. However, traditional conformal inference requires a dedicated calibration dataset, imposing a "data tax" that reduces the amount of data available for model training. This paper introduces Conformalized Early Stopping (CES), a novel framework that allows practitioners to reuse the same validation data for both early stopping and conformal calibration. By employing a test-augmented model selection rule, CES maintains finite-sample statistical guarantees without requiring an additional data split, resulting in more powerful models and tighter prediction intervals.
Introduction
Deep learning models have achieved state-of-the-art performance across various domains, yet they remain prone to "hallucinating" certainty even when incorrect. Conformal inference has emerged as a popular solution, providing a framework to generate prediction sets—such as a list of likely labels in classification or an interval in regression—that contain the true value with a user-specified probability (e.g., 95%).
A significant bottleneck in applying conformal inference is the requirement for a hold-out "calibration set." In standard workflows, practitioners already split their data into training and validation sets, using the latter for early stopping to prevent overfitting. Adding a third split for calibration further depletes the training pool, which is particularly detrimental in data-constrained environments. This paper addresses the question: Can we use the same hold-out data for both early stopping and conformal calibration without violating the statistical assumptions required for validity?
Methodology: Conformalized Early Stopping (CES)
The core innovation of Conformalized Early Stopping (CES) is a "test-augmented" model selection rule. Standard early stopping typically selects a single model $M$ that minimizes loss on a validation set $\mathcal{D}_{es}$. If one were to use this same $\mathcal{D}_{es}$ for conformal calibration, the resulting guarantees would be invalid because the model $M$ was specifically chosen to perform well on that exact data.
CES bypasses this dependency by including the test point $(X_{n+1}, y)$ in the selection process in an exchangeable manner. The process follows these primary steps:
- Snapshot Storage: During the training phase, the algorithm saves snapshots of the model weights at regular intervals (e.g., every few epochs).
- Augmented Loss Calculation: For a given test point $X_{n+1}$, the algorithm considers every possible candidate label $y$. It calculates an augmented loss that combines the error on the validation set with the hypothetical error on the test point, assuming $y$ is the correct label.
- Dynamic Model Selection: Instead of using a single global model, CES selects a potentially different "best" snapshot for every possible label $y$ (in classification) or across the range of $y$ (in regression).
- Conformal Calibration: The standard conformal machinery is then applied using the dynamically selected model.
For regression tasks, where $y$ is continuous, the authors developed a "divide-and-conquer" algorithm to efficiently find the lower envelope of loss parabolas. This ensures the computational overhead remains manageable, with an algorithmic complexity of $O(T \log T)$ where $T$ is the number of snapshots.
Experimental Evaluation
The researchers evaluated the performance of CES against standard data-splitting benchmarks across several domains:
- Computer Vision: Multi-class classification and outlier detection using the CIFAR-10 dataset.
- Regression: Various UCI Machine Learning repository datasets, including protein physicochemical properties (bio), bike rental counts (bike), and compressive strength (concrete).
- Quantile Regression: Experiments conducted on datasets such as MEPS, STAR, and community data.
Key Results
1. Statistical Validity and Coverage Across all experiments, CES consistently met or exceeded the nominal coverage levels (e.g., 90%). This confirms that the "recycled" validation data does not break the statistical guarantees of the conformal framework, provided the test-augmented selection rule is used.
2. Efficiency and Predictive Power By eliminating the need for a third data split, CES allows more data to remain in the training set. In regression tasks on the bio dataset, CES produced significantly shorter (narrower) prediction intervals than models trained with standard data splitting. For example, at a sample size of 2000, the efficiency gains from a larger training set were clearly visible in the tighter intervals.
3. Superior Classification and Outlier Detection In CIFAR-10 classification, CES achieved the smallest "cardinality" (the number of labels included in a prediction set) compared to benchmarks. In outlier detection, CES maintained the target False Positive Rate (FPR) of 10% while achieving a significantly higher True Positive Rate (TPR) than traditional methods.
Discussion and Limitations
While CES offers a "free" path to conformal inference, it introduces minor overheads. Storing approximately 100 snapshots of a model is generally inexpensive in modern cloud environments, but it could pose a storage bottleneck for exceptionally large architectures, such as massive transformers. Additionally, the dynamic selection process in regression is more computationally involved than simple point-prediction pipelines, though the authors' optimized algorithm mitigates this.
Conclusion
Conformalized Early Stopping demonstrates that the "data tax" traditionally associated with rigorous uncertainty quantification is optional. By integrating model selection and calibration into a single, exchangeable process, CES enables the use of more powerful models with tighter, statistically valid prediction sets. This approach paves the way for more efficient machine learning pipelines, with potential future applications in hyperparameter tuning and feature selection.
Summary updated