Calibrating the Ensemble You Actually Deploy, Without a Holdout

When an ensemble is built by training each member on a different random subset of the data, every training point is automatically unseen by some members. Those "out-of-bag" points can stand in for a validation set, so no labeled data has to be set aside. The catch, which this proposal targets, is that the out-of-bag prediction for a point averages only the few members that did not see it. The deployed model averages all of them. Averaging more members makes predictions less extreme and changes how well calibrated they are. Out-of-bag estimates therefore describe a smaller ensemble than the one shipped. For random forests with hundreds of trees the gap is negligible. For deep ensembles with five to twenty networks, which is exactly where setting aside validation data hurts most, it is large enough to pick the wrong temperature, the wrong weight decay, or the wrong stopping epoch.

We propose an estimator that corrects out-of-bag calibration statistics for ensemble size. It extrapolates from the sub-ensembles that did not see each point to the full ensemble. For the Brier score the correction follows exactly from a variance identity. For log-loss and calibration error it is an extrapolation in $1/m$, where $m$ is the number of averaged members. Paired with structured subsampling designs that guarantee every point several out-of-bag members, the estimator lets a small deep ensemble jointly tune its regularization and calibration on 100% of the labeled data. The goal is to match or beat the same ensemble tuned on a dedicated holdout.

Core Claim

Two observations motivate the claim. The first is that conditional on point $i$ being excluded, an out-of-bag member is distributed exactly like a deployed member evaluated on a fresh test point, since both were trained on a random fraction of data that does not contain the point being scored, up to a dataset of size $n-1$ versus $n$. So the only systematic mismatch between out-of-bag evaluation and test-time behaviour is ensemble size. The second is that ensemble size enters the Brier score in closed form. Let $\bar p_m(x)$ be the average predicted probability vector of $m$ independently trained members, $\mu(x)$ its expectation over training randomness, $\sigma^2(x)$ the summed per-class variance of a single member's prediction, and $y$ the one-hot label. Then

$$ \mathbb{E}\,\|\bar p_m(x)-y\|^2 \;=\; \mathbb{E}\,\|\mu(x)-y\|^2 \;+\; \frac{1}{m}\,\mathbb{E}\,\sigma^2(x), $$

which holds because the members are independent given the data. Point $i$ has $m_i \ge 2$ out-of-bag members. From them we estimate $\sigma^2(x_i)$ unbiasedly by the sample variance and correct the out-of-bag Brier score to the deployed size $B$ by subtracting $(1/m_i - 1/B)\,\hat\sigma^2_i$. For negative log-likelihood and expected calibration error, which have no exact decomposition, we evaluate every sub-ensemble size from 1 to $m_i$ among each point's out-of-bag members and fit the statistic as a low-order function of $1/m$. We then read it off at $1/B$. The claim to be tested is that this corrected estimate ranks hyperparameter configurations of the deployed ensemble as well as a holdout of comparable size. Because no data is withheld, it should then deliver equal or better test calibration at equal compute, with the advantage growing as the labeled set shrinks.

Relation to Prior Work

The closest work is On Joint Regularization and Calibration in Deep Ensembles. It shows that weight decay, temperature and early stopping should be tuned for the ensemble as a whole rather than per member, the "ensemble optimality gap". It avoids a global holdout by giving each member a partially overlapping validation split and evaluating pairs of members jointly as a proxy for the ensemble. That overlapping design is structurally an out-of-bag design. We take it as the motivating result and as the main competitor. Our departure is in the estimator: a pairwise proxy scores a two-member ensemble, while we explicitly correct to the $B$-member ensemble. We also design the exclusion pattern so the correction is estimable.

A second line uses out-of-bag points to remove calibration holdouts for interval prediction. Predictive inference is free with the jackknife+-after-bootstrap aggregates the models that excluded each point to obtain leave-one-out residuals from $B$ fits. Nested conformal prediction and quantile out-of-bag ensemble methods does the same with quantile forests and shows it beats split conformal when $n \le 100$. Stability of Random Forests and Coverage of Random-Forest Prediction Intervals proves near-nominal coverage for out-of-bag intervals under forest stability. These works supply the core mechanism of scoring each point with the members that excluded it. They also set a constraint: they need enough members that every point is out-of-bag somewhere, which is exactly what fails for small deep ensembles under plain bootstrap. None of them addresses the size mismatch, because their target is coverage of residual quantiles in large forests rather than probability calibration of small ensembles.

On the classification side, Random forest calibration uses out-of-bag predictions to fit Platt scaling, isotonic regression and Venn-Abers on 30 tabular datasets, and finds an intermediate tree depth that minimizes calibration error on its own. Hyperparameters and Tuning Strategies for Random Forest tunes forests on out-of-bag error and reports average gains of 0.014 misclassification error and 0.007 AUC over defaults on 39 OpenML datasets. Both operate where $B$ is in the hundreds. Our analysis predicts the correction will be small there, so they serve as a negative control rather than a target.

Holdout-free alternatives outside ensembles frame the competition. Hyperparameter Optimization through Neural Network Partitioning shards weights and data so subnetworks are scored on unseen shards, reaching 89.1% CIFAR-10 accuracy. It optimizes point-prediction loss, not calibration, and reduces active capacity. Proximal Validation Protocol replaces the holdout with selected augmented samples and reports a 32.3% average reduction in the test-validation gap. Its validation data are synthetic, so it inherits whatever distribution shift augmentation introduces. Making Learners (More) Monotone documents the cost of the alternative it improves on: gating decisions with validation data works, but the data spent on validation directly reduces sample efficiency.

Research Program

The work has three parts. First, we derive the size-corrected estimators. That means the exact Brier correction above, and a $1/m$ extrapolation for log-loss and for expected calibration error, including the top-label variant studied in Top-label calibration. For each we characterize the bias from the $n-1$ versus $n$ training-set size and the variance as a function of $m_i$. Temperature is handled by computing the corrected log-loss on a grid of temperatures applied to the averaged prediction and choosing the minimizer.

Second, we build balanced exclusion designs. The data are split into $K$ folds, and each of $B$ members excludes a fixed number of folds so that every point has exactly $m$ out-of-bag members while each member trains on a chosen fraction $q$. For example, $B=10$ with each member excluding 2 of 10 folds gives $q=0.8$ and $m=2$, while $B=20$ excluding 2 of 20 gives $q=0.9$ and $m=2$. This replaces the bootstrap, which with five members leaves about a third of points with no out-of-bag member at $q=0.8$. It makes explicit the trade-off between per-member data, which favors high $q$, and estimator variance, which favors high $m$.

Third, we use the corrected estimates inside the joint tuning loop of Fredsgaard and Schmidt for weight decay, temperature and early stopping, so the comparison changes only the evaluation signal.

Baselines

The naive reference is a deep ensemble trained on all data with random initializations and no calibration. It costs nothing in data but is known to be miscalibrated. Top-label calibration reports top-label calibration error typically 2-3 times standard confidence calibration error across vision models. Any proposal must improve on it in log-loss and calibration error without losing more than noise-level accuracy.

The conventional baseline is a global holdout. Every member trains on the same 80% or 90%, and joint tuning plus temperature scaling uses the remaining split. It is run with the same members, the same grid and the same number of training runs. Beating it means lower test log-loss and calibration error at equal $B$ and equal compute, with no accuracy loss. Because the holdout baseline has the same per-member data fraction as our $q=0.8$ design, a win there isolates the value of collectively seeing all data plus correct estimation.

The strongest existing method is the overlapping-holdout strategy of On Joint Regularization and Calibration in Deep Ensembles. We reimplement it with the authors' pairwise joint evaluation on CIFAR-10, CIFAR-100, AG News, NCI1 and covertype, the benchmarks it uses. We hold their split pattern fixed, and the identical members are evaluated with their proxy and with ours. No reported result found on Lacuna for the numerical log-loss or calibration error of this method on any benchmark, so we reproduce it ourselves before comparing. Beating it means better selection regret, defined as the test log-loss of the chosen configuration minus that of the oracle-best configuration, and better final calibration on the same trained models.

The key ablation is uncorrected out-of-bag evaluation on the same exclusion design. Any improvement over it is attributable to the size correction alone.

We also include Proximal Validation Protocol as a synthetic-validation competitor on CIFAR-10-LT and CIFAR-100-LT, where it reports its 32.3% gap reduction. We further include an oracle that tunes on a large test-distribution set, giving the ceiling.

Why not just…?

Why not just use many more members so the out-of-bag ensemble is already large? For forests this is the right answer, and our tabular control checks that the correction vanishes there. For deep networks, each extra member is a full training run. The regime where validation data are scarce is usually the regime where compute for 100 networks is also absent. The experiments fix $B$ between 5 and 20 to reflect this, and report how the uncorrected estimator's bias shrinks as $B$ grows, so a reader can see at what size the correction stops mattering.

Why not just hold out 10% and temperature-scale? With tens of thousands of labels this may be nearly as good, and we expect it to be competitive on full CIFAR-10. The evidence against it is concentrated at small $n$. Nested conformal prediction and quantile out-of-bag ensemble methods finds split calibration clearly worse at $n \le 100$. The Pick-to-Learn Algorithm reports actual risk of 0.173 for a test-set-split approach versus 0.064 for its holdout-free method at $N=200$. We therefore run labeled-subset sweeps (CIFAR-10 at 1k, 5k and 50k labels; covertype subsamples from 500 to full). If the holdout baseline ties us at every size, the proposal's value reduces to removing a tuning knob, and we will report it that way.

Why not just use the overlapping holdout as published? This is the real threat, because the data layout is nearly the same as ours. The difference is what gets estimated: a pairwise evaluation scores two-member averages, whose confidence and calibration differ systematically from a $B$-member average. The same-models comparison described above settles it directly. If pairwise and corrected estimates select the same configurations, our contribution is limited to the analysis and the Brier identity. If they diverge, the test-set oracle tells us which is right.

Proposed Experiments

The first decisive check measures estimator fidelity. On each benchmark we train ensembles across a grid of weight decay, stopping epoch and temperature. We then compare corrected and uncorrected out-of-bag estimates, and a 10% holdout estimate, against the true test Brier score, log-loss and top-label calibration error of the deployed $B$-member ensemble. Fidelity is reported as absolute error, rank correlation over the grid, and selection regret. The prediction is that uncorrected estimates are biased toward overconfidence-penalizing configurations, choosing temperatures that are too low, because small sub-ensembles are sharper. The correction should remove most of this bias at $m=2$ to $4$.

The second check is end-to-end. Test log-loss, calibration error and accuracy are measured on CIFAR-10/100 with Wide-ResNet-16-4, AG News with a BiLSTM, NCI1 with a graph convolutional network, and covertype with an MLP, at matched compute, across the labeled-subset sweep.

The third check is a sweep over the per-member fraction $q$ and out-of-bag count $m$. It identifies the best design, testing whether the correction makes high-$q$ designs ($q=0.9$) usable that uncorrected out-of-bag evaluation cannot support.

The fourth check is the negative control on the 30 UCI and PROMISE datasets of Random forest calibration, with $B=500$ trees. There the correction should change selected depths and Venn-Abers fits by no more than noise. A large effect there would indicate the correction is fitting noise rather than ensemble size.