A single training run already passes through many different models, the saved snapshots or "checkpoints" along the way. Two communities treat these checkpoints in opposite ways. One averages or extrapolates them into a single better model, which costs nothing extra at inference. The other keeps them as an ensemble and averages their predictions, which gives better uncertainty but multiplies inference cost and storage by the number of members. We propose that the difference between checkpoints has two parts that should be handled separately. The first is progress along the direction the optimizer is travelling. That part should be collapsed into one model, because earlier checkpoints are simply worse. The second is sideways wobble orthogonal to that direction. That part is where useful disagreement lives, and it is low-dimensional enough to store compactly and to evaluate cheaply as a small committee confined to the network's last blocks.
Concretely, we would stream a low-rank summary of the late-training trajectory instead of saving full checkpoints. We would build one merged model from the dominant direction. We would then construct committee members that sit at the merged point and differ from it only by each checkpoint's sideways residual. The target is ensemble-like calibration and out-of-distribution detection at roughly 1.2 to 1.3 times single-model inference cost, with only a few extra copies of the last blocks stored.
Core Claim
The unresolved tension this addresses is stated plainly in the literature. Checkpoint Ensembles harvests the top-$k$ checkpoints of one run and averages their predictions. It explicitly argues that averaging their weights fails, but it pays $k$ forward passes and stores $k$ full models. Learning Neural Network Subspaces trains a line or simplex of models and finds that the midpoint beats standard training at 1× cost, while ensembling the endpoints in output space still gives strictly better accuracy and calibration. Extra-Merge shows that late pre-training trajectories collapse onto a nearly one-dimensional "river valley", where the first principal component explains over 94% of checkpoint variance. It exploits this only for single-point extrapolation and never asks what the remaining variance is good for.
Our claim is that the collapse-versus-ensemble debate conflates these two components. A raw checkpoint ensemble mixes progress along the valley with disagreement across it, and the progress component hurts: it drags in less-trained members. That is why Checkpoint Ensembles must select only top-$k$ snapshots. Recentering removes the progress component. Let $\theta_1,\dots,\theta_K$ be late-training checkpoints restricted to the last $L$ blocks of the network, $\bar\theta$ their mean, $u_1$ the leading principal direction, $P_\perp$ the projection onto the next $d$ principal directions, and $\theta^\star$ the merged (averaged or Extra-Merge-extrapolated) model. Committee member $k$ is
$$\hat\theta_k = \theta^\star + \lambda\, P_\perp(\theta_k - \bar\theta),$$
where $\lambda$ is a single scale fitted on held-out data. The network trunk before the last $L$ blocks is shared and evaluated once. Storage is $d+1$ vectors of last-block size plus $K$ small coefficient vectors. We compute them with streaming principal component analysis, so no window of full checkpoints is ever held on disk. This directly removes the $K$-checkpoint storage burden that Extra-Merge and Trainable Weight Averaging carry.
Two prior results motivate confining the committee to late blocks. Calibrating a Deep Neural Network with Its Predecessors shows that deeper blocks overfit and become overconfident earlier, so historical versions of those blocks carry calibration signal. Late-phase weights shows that replicating a small parameter subset late in training yields useful diversity. The mechanism differs from theirs in two ways. Unlike the predecessor search, we perform no combinatorial search and produce an actual committee rather than one stitched model. Unlike late-phase weights, we train no replicated weights at all, because the members come for free from the trajectory.
The design is also constrained by Is BatchEnsemble a Single Model?. That paper finds that trained rank-1 member perturbations collapse to near-identical members (cosine similarity near 1.0) and give no OOD benefit. Our members are not trained to differ, so they cannot collapse through a shared gradient signal. Whether their residuals are functionally diverse enough is an empirical question the experiments must answer, not an assumption.
Research Program
The first step is measurement. On image classifiers, we would record the spectrum of the late trajectory in the last blocks under standard step decay, cosine, and cyclic learning rates. Extra-Merge's rank-1 finding comes from LLM pre-training, and it is not known whether classifier trajectories share it. For each spectral component we would measure two things: how much prediction disagreement it induces on in-distribution, corrupted, and OOD inputs, and how much it changes validation loss. The hypothesis predicts that the leading direction mostly moves loss while the residual directions mostly move disagreement, concentrated on shifted inputs. If disagreement instead lives in the leading direction, the premise fails and we would report that.
The second step is construction. We would implement streaming principal component analysis over checkpoints saved every few hundred steps in the final 25–40% of training. We would compute $\theta^\star$ both as a uniform average and as an Extra-Merge extrapolation, and we would sweep the three design choices: $L$ (number of committee blocks), $d$ (residual rank), and $K$ (number of members). The object reported is the Pareto frontier of calibration and OOD detection against inference FLOPs and stored bytes.
The third step is a transfer test in the regime where the valley is known to exist. On GPT-2 124M trained on the FineWeb 10B subset, the setting of Extra-Merge, we would ask whether a recentered committee over the last transformer blocks improves token-level calibration and zero-shot accuracy on ARC and HellaSwag over the Extra-Merge point alone. This also probes whether the late-block approach survives in LayerNorm architectures, where late-phase BatchNorm replication does not apply.
Baselines
The naive reference is a single model at the end of standard training, with and without temperature scaling. On CIFAR-10, Is BatchEnsemble a Single Model? reports 94.1% accuracy, SVHN OOD AUROC of 0.893, and CIFAR-10-C ECE of about 0.323 for the single model.
The zero-storage weight-space competitor is the SWA or merged point $\theta^\star$ plus temperature scaling. It uses the same run and the same held-out calibration set. It is the direct test of whether the committee adds anything beyond the merged model.
The accuracy and uncertainty ceiling is a Deep Ensemble of independently trained networks, at 4× or 5× the cost. The same paper reports 95.2% accuracy, 0.924 SVHN AUROC, and about 0.206 CIFAR-10-C ECE.
The strongest single-cost competitor is LatentBE from Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation. It distills an ensemble into a BatchEnsemble student and collapses the student to one model. On CIFAR-100 it reports 80.02% accuracy, NLL of 0.792, and 0.107 ms/img, against 0.390 ms/img for a 4-model ensemble. It needs a trained teacher ensemble, so we would report its total training compute alongside its accuracy.
For calibration specifically, the predecessor search reports CIFAR-100 ResNet-110 ECE of 1.76%. The late-phase-weights non-averaged ensemble reports SVHN AUROC of 0.877 against a 0.814 baseline in its own setup. We would rerun both rather than compare across setups.
Two further baselines are needed. The first is the output-space Checkpoint Ensemble with top-$k$ full checkpoints, for which no reported numerical result was found on Lacuna for CIFAR. The second is MC-Dropout at matched pass count.
All methods share architectures (ResNet-32 and WRN-28-10 on CIFAR, ResNet-50 at ImageNet scale if budget allows), data, total training epochs, and an equal hyperparameter search budget of about 20 trials each.
The decisive ablation is the un-recentered late-block committee. It uses the same $L$, $K$, and storage, but members are raw late-block snapshots with no projection and no recentering. It isolates the decomposition. A second ablation uses random orthogonal directions of matched norm in place of trajectory residuals. It tests whether trajectory structure matters or any perturbation would do.
Beating the field means the following. At no more than 1.3× single-model FLOPs, the committee should close at least half of the single-to-Deep-Ensemble gap on CIFAR-10-C ECE and SVHN AUROC. It should also significantly exceed both the SWA-plus-temperature point and the un-recentered committee across three seeds.
Why not just…?
Why not just run SWA, an exponential weight average, or Schedule-Free AdamW and skip storage entirely? Through the River shows that Schedule-Free AdamW gets averaging benefits without an extra weight copy, and SWAD shows that flat-minimum averaging gives +3.6pp over ERM on DomainBed. These methods still produce one point. The single-versus-ensemble gaps in CIFAR-10-C ECE (0.323 vs 0.206) and SVHN AUROC (0.893 vs 0.924) are exactly what a single point does not close. Our $\theta^\star$ is such an average, so the comparison against it is built in. If the committee adds nothing over $\theta^\star$ under shift, the idea fails.
Why not just apply temperature scaling or Mix-n-Match to one model? Mix-n-Match calibrates well in-distribution with little data. This is a real threat on in-distribution ECE, and we expect it to tie there. The predecessor-search paper reports that temperature adjustments do not fix overconfidence under shift and on OOD inputs. We therefore judge primarily on CIFAR-10-C and SVHN, and we apply temperature scaling to every method, including ours.
Why not just keep a few raw late checkpoints of the last blocks as a head ensemble? This is the most serious alternative, because it costs the same as our method. It is exactly the un-recentered ablation. If it matches the recentered committee, the decomposition is unnecessary. The honest contribution would then shrink to the storage result from streaming compression plus the finding that late-block snapshot ensembles suffice. The experiments are built so that this outcome is visible rather than hidden.
Why not just distill a Deep Ensemble? Standard distillation loses diversity. LatentBE reports NLL of 1.038 for standard knowledge distillation against 0.792 for its own method. Both require training the teacher ensemble first, which multiplies training cost. We compare against LatentBE at equal inference cost and report training FLOPs, where our method should cost essentially one run.
Decisive Checks
The idea stands or falls on three results. First, the spectral analysis must show that residual directions, not the leading one, carry disagreement on shifted inputs. Second, the recentered committee must beat the un-recentered committee and the random-direction control at matched storage and FLOPs. Third, it must beat the merged model with temperature scaling on CIFAR-10-C ECE and SVHN AUROC across seeds while staying at or below 1.3× inference cost. The GPT-2 experiment then tests whether the effect survives in the setting where the one-dimensional valley was discovered.
References
- Checkpoint Ensembles: Ensemble Methods from a Single Training Process
- Learning Neural Network Subspaces
- Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
- Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions
- Calibrating a Deep Neural Network with Its Predecessors
- Neural networks with late-phase weights
- Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles
- Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training
- SWAD: Domain Generalization by Seeking Flat Minima
- Mix-n-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning