Does Joint Ensemble Tuning Survive Honest Validation?

Ensembles average the predictions of several models, and a growing body of work argues that their members should be tuned together, for the ensemble's benefit, rather than each tuned alone. A member that stops training later than its own best epoch, or a hyperparameter setting that looks mediocre in isolation, can make the group better. The catch is that every one of these collective decisions is scored on the same held-out validation set. Tuning M members jointly gives the search roughly M times more freedom to fit that set's noise, and the cases where ensembles matter most, small datasets, are exactly the cases where the validation set is smallest. We propose to measure how much of the reported benefit of collective tuning is real improvement and how much is extra validation overfitting, under budget-matched, leakage-free comparisons. We also propose a cheap, data-free test that tells a practitioner which collective knobs are safe to tune at a given validation size.

The core prediction is simple. Collective tuning of low-dimensional knobs, such as one shared stopping epoch or one shared temperature, should keep its gains even with small validation sets. Collective search over per-member configurations should lose its advantage below some validation size, and that threshold should grow with ensemble size and search budget. If the prediction holds, the field gets a usable rule and a corrected picture of the "ensemble optimality gap". If it fails, we learn that collective tuning is safer than the theory of validation leakage suggests, which is equally worth knowing.

Core Claim

The ensemble optimality gap is the extra accuracy available from optimizing members for the group instead of for themselves. Two papers supply the main evidence for it, and this proposal builds on both. Bayesian Hyperparameter Optimization for Ensemble Learning runs a round-robin Bayesian search, called Ensemble Optimization, that scores each candidate by the validation loss of the whole ensemble with that candidate swapped in. It reports an average rank of 1.89 against 3.39 for the best single model. On Joint Regularization and Calibration in Deep Ensembles tunes weight decay, early stopping and temperature jointly across deep ensemble members. It finds that joint early stopping consistently helps, and it introduces an "overlapping holdout" so that less data is withheld.

Neither paper asks whether the validation-measured gain is inflated by the search itself. Two other papers constrain how this should be read. Unbiased Assesment of Learning Algorithms treats tuning as a channel that leaks label information from the evaluation data and shows rankings can be distorted after as few as 18 trials. Stability and Generalization of Bilevel Programming in Hyperparameter Optimization bounds the validation-to-test gap of gradient-based tuning as growing with outer iterations and shrinking with validation size $m$.

Our claim is that the apparent collective gain should be decomposed explicitly:

$$\Delta_{\text{test}} = \Delta_{\text{val}} - \big(O_{\text{joint}} - O_{\text{indiv}}\big)$$

The terms are defined as follows. $\Delta_{\text{val}}$ is the validation-loss improvement of the jointly tuned ensemble over the independently tuned one. $\Delta_{\text{test}}$ is the same improvement measured on untouched test data. $O_{\text{joint}}$ and $O_{\text{indiv}}$ are the optimism of each procedure, meaning how much better its selected configuration looks on validation than on test.

We hypothesize that the extra optimism of joint tuning, $O_{\text{joint}} - O_{\text{indiv}}$, scales with the effective number of free collective choices. By that we mean the dimension of the joint knob times the number of trials. We also hypothesize that it shrinks with $m$. This direction follows from the leakage argument and the stability bound, though the exact rate for ensembles is what we measure rather than assume.

Research Program

The work has three parts.

A budget-matched evaluation protocol. The first part builds a protocol that removes the benchmarking confounds documented elsewhere. On the State of the Art of Evaluation in Neural Language Models showed that claimed architectural gains vanished once baselines received equal tuning budgets, with a well-tuned LSTM reaching 58.3 test perplexity on Penn Treebank against 62.4 for a NAS cell. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline showed that picking the best epoch using test feedback, along with other training tricks, inflated point-cloud results. Our protocol carries these lessons over to ensembles. Every method gets the same number of trained networks, the same total training epochs, the same data split, and the same parameter count. Test data is touched once.

Tracing the optimism-versus-gain curve. The second part traces this curve across a ladder of collective knobs of increasing dimension. The ladder runs from a shared temperature, to a shared stopping epoch, to per-member stopping epochs, per-member weight decay, and finally full per-member configuration search in the style of Ensemble Optimization. Each rung is crossed with validation size, ensemble size and trial count. Total data is held fixed while the validation fraction varies, so the validation tax is part of the measurement rather than ignored.

A data-free split-half reversal test. The third part builds and validates a cheap diagnostic that needs no extra data. The validation set is split in half. The collective selection is run on one half and scored on the other, and the roles are then swapped. The rate at which the joint choice's advantage reverses, together with the size of its drop, estimates its optimism. A practitioner would adopt a collective knob only when its validation gain exceeds this estimated optimism. We will check whether this rule predicts the true test outcome measured by the full protocol.

Proposed Experiments

We reuse the four settings of On Joint Regularization and Calibration in Deep Ensembles so that results are directly comparable to the strongest existing joint-tuning evidence. These are CIFAR-10 with Wide ResNet-16-4, Covertype with an MLP, NCI1 with a graph network, and AG News with a BiLSTM. We create data-scarce regimes by subsampling the training pool, for example to 1%, 5% and 25% of each dataset, which is where validation overhead bites.

For the two small tabular and graph settings, full nested cross-validation is affordable. It serves as the ground-truth estimate of each procedure's real performance and optimism. On CIFAR-10, a large reserved test pool plays that role. Metrics are classification error, negative log-likelihood and expected calibration error of the ensemble. We also report the measured optimism of each procedure and the agreement between the split-half estimate and the nested-CV optimism. Ensemble sizes are 3, 5 and 10. Trial budgets range from tens to a few hundred configurations, and each cell is repeated over many random splits so that optimism differences come with confidence intervals.

Baselines

Independent tuning (the do-nothing reference). The naive reference is the standard deep ensemble. Each member is tuned and early-stopped on its own validation loss, and predictions are averaged. It receives exactly the same number of trials and trained networks as every joint method, split evenly across members. It is the reference because Joint Training of Deep Ensembles Fails Due to Learner Collusion shows independent members can be very hard to beat. In that paper, independently trained ResNet-18 ensembles reached 68.10% ImageNet top-1, while joint end-to-end training reached 62.74%, because members learned to cancel each other's extreme outputs.

Joint early stopping and temperature scaling with the overlapping holdout. This is the strongest existing method for low-dimensional collective tuning, from On Joint Regularization and Calibration in Deep Ensembles. We run it with that paper's holdout scheme on identical splits. No reported result was found on Lacuna for its exact error, likelihood or calibration numbers on these datasets, so we reproduce it ourselves under our protocol.

Ensemble Optimization. This is the strongest method for high-dimensional collective search, from Bayesian Hyperparameter Optimization for Ensemble Learning. Its reported figure is an average rank of 1.89 against 3.39 for the best single model, measured on validation-selected configurations across its benchmark search spaces. Beating it, in our sense, does not mean a lower validation loss. It means showing that on budget-matched, once-touched test data, either its advantage persists only above a validation size our diagnostic predicts, or a rule that adopts it only when the diagnostic allows does at least as well on test at every validation size.

Ablation of the key ingredient. The ablation isolates the proposal's key ingredient, the dimensionality of the collective decision. We hold the search method fixed and vary only the knob dimension along the ladder. If optimism tracks dimension and validation size as predicted, the decomposition explains the field's mixed results. If it does not, the core claim is wrong.

Why not just…?

Why not just use nested cross-validation? Unbiased Assesment of Learning Algorithms recommends nested resampling, and where it is affordable it is the right answer; we use it as ground truth for exactly that reason. The objection is a real threat on small tabular data. Our contribution there is the measurement of the gap, not a replacement. For deep ensembles the cost multiplies the number of networks trained by both fold counts on top of the ensemble size, which is what the file identifies as intractable. The split-half test costs only extra evaluations, not extra training. The experiment that settles this is whether split-half estimates match nested-CV optimism on Covertype and NCI1. If they do not, our diagnostic has no case.

Why not just use a larger validation set? In the subsampled regimes, every point moved to validation is taken from training. Making Learners (More) Monotone documents how withholding data hurts sample efficiency and can make learning non-monotone. Our fixed-total-data sweep directly tests whether enlarging validation to make joint tuning safe costs more test accuracy than joint tuning gains. If it does not, the simple advice wins and we will report it.

Why not just bag the members and use out-of-bag points for collective decisions? If each member trains on a different subset, every training point is unseen by some members. This yields holdout-free scores, much as Hyperparameter Optimization through Neural Network Partitioning obtains out-of-sample signal within one training run, reporting 89.1% versus 79.5% for standard training on CIFAR-10. The drawback is that out-of-bag predictions come from sub-ensembles, not the full ensemble the collective objective targets, and each member sees less data. That is the same capacity penalty the partitioning paper incurs. We include an out-of-bag variant of joint early stopping as an additional arm. If it matches the overlapping holdout at equal total data, it becomes the recommended low-overhead scheme, and our optimism analysis applies to it unchanged.

Why not just add a diversity penalty during training instead of tuning? Diversity regularization in deep ensembles lowers CIFAR-100 calibration error from 4.3% to 2.5% with a negative-correlation penalty. However, Joint Training of Deep Ensembles Fails Due to Learner Collusion shows such penalties can be satisfied by fake diversity, such as constant offsets, and that members become fragile when any one is dropped. The penalty weight is itself a collective hyperparameter chosen on validation, so it sits on our ladder rather than outside it. We include it there and run the drop-one-member test to check for collusion.

Decisive Checks

The proposal succeeds if three things hold. First, measured extra optimism of joint tuning grows with knob dimension and shrinks with validation size, with confidence intervals that exclude zero in the small-data regimes. Second, low-dimensional joint stopping and temperature keep a positive test gain at sizes where per-member search does not. Third, the split-half rule's adopt-or-reject decisions agree with nested-CV ground truth well above chance. It fails if, under budget-matched protocols, high-dimensional joint search beats independent tuning on test at every validation size. In that case, collective tuning is robust to validation reuse, and the leakage concern can be set aside for ensembles.