Distilling Into the Center: Online Ensemble Distillation That Deploys as One Model

A common way to get ensemble-quality predictions from a single network takes two expensive steps. First you train several large models, the ensemble. Then you train a small "student" model to imitate their averaged predictions, a process called knowledge distillation. The cheaper alternatives each give up something. Online distillation trains peer models together so they teach each other, but at the end it throws all but one of them away, and the peers tend to become copies of each other. Weight averaging is the other route: one model whose weights are the average of several related models. It runs at single-model cost, but in the best existing version the models being averaged are distilled from an ensemble that was trained beforehand.

We propose to train a small family of models that lie on a shared low-loss region of weight space, a line or triangle whose corners are separate weight vectors. The corners' combined predictions serve as an online teacher. That teacher is distilled only into the center of the region, the average of the corners, which is the single model we deploy. The design rests on one geometric fact. A loss evaluated at the average of several weight vectors pushes every corner by exactly the same amount. Pulling the center toward the consensus therefore moves the whole region rigidly and cannot shrink its spread. The consensus pressure that makes online peers collapse into copies is, by construction, orthogonal to the diversity that makes the teacher worth having. The result would be one training run, no pre-trained teacher, and one deployed network.

Core Claim

Let $\omega_1,\dots,\omega_m$ be the corner weight vectors and let $\theta_c = \frac{1}{m}\sum_i \omega_i$ be their average, the deployed model. Training follows Learning Neural Network Subspaces, which trains such a region from scratch. At each step it samples mixing weights $\alpha$, evaluates the network at $\theta_\alpha=\sum_i\alpha_i\omega_i$ with cross-entropy, and adds a squared-cosine penalty that keeps the corners apart. We add one term:

$$ \mathcal{L}_{\text{center}} = \lambda(t)\,\mathrm{KL}\big(\mathrm{sg}[\bar p(x)] \,\|\, p_{\theta_c}(x)\big), \qquad \bar p(x)=\tfrac{1}{m}\textstyle\sum_i p_{\omega_i}(x). $$

Here $\bar p$ is the averaged prediction of the corner networks. $\mathrm{sg}$ stops gradients from flowing into the teacher. $\lambda(t)$ is a weight ramped up from zero, because Asymmetric Decision-Making in Online Knowledge Distillation shows that immature online teachers mislead students early in training.

Because $\partial\theta_c/\partial\omega_i = 1/m$ for every $i$, the gradient of this term with respect to each corner is the same vector, $\frac{1}{m}\nabla\mathcal{L}_{\text{center}}(\theta_c)$. Under SGD with momentum it leaves the spread $\sum_i\|\omega_i-\theta_c\|^2$ exactly unchanged. Under adaptive optimizers such as Adam, which rescale each corner's gradient separately, the invariance holds only approximately.

The claim to be tested is that this center-only distillation improves the deployed center model over plain region training. It should also beat diversity-enhanced online distillation at equal inference cost, and match the offline two-stage pipeline at a fraction of its training compute.

Relation to Prior Work

Several papers motivate the gap. Collaborative learning (Collaborative Learning for Deep Neural Networks) and online ensemble compression (Online Ensemble Model Compression Using Knowledge Distillation) train peer heads or branches on a shared backbone in one stage, then deploy a single branch and discard the rest. KDCL (Online Knowledge Distillation via Collaborative Learning) does the same with separate peer networks. Two papers show that such peers converge toward identical states and push back with extra machinery. Multi-branch diversity enhancement (Online Knowledge Distillation via Multi-branch Diversity Enhancement) orthogonalizes the classifier heads, and OKDDip (Online Knowledge Distillation with Diverse Peers) uses asymmetric attention between peers. Both still distill into a separate "group leader" network and discard the diverse peers. We differ on two points: the deployed model is the weight average of the peers rather than a separate network, and we remove the source of collapse by geometry rather than adding a counter-force against it.

Other papers constrain the design. Is BatchEnsemble a Single Model? finds that BatchEnsemble members, which differ only by small per-member scaling vectors, reach weight cosine similarity near 1.0 and lose nearly all disagreement. We take this as a warning: we must measure corner spread and disagreement directly rather than assume them. Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (LatentBE) is the closest existing method. It shows that student subnetworks kept in one low-loss valley can be averaged into a single fast model, but it requires a fully pre-trained deep ensemble as teacher. We keep its "average at deployment" idea and drop the pre-trained ensemble.

Two papers supply the method. The subspace paper supplies the training of a low-loss region and its cosine penalty, but it uses no distillation. Averaging Weights Leads to Wider Optima and Better Generalization (stochastic weight averaging, SWA) supplies the evidence that averaging weights inside a flat basin approximates averaging predictions.

Research Program

We will build the method on top of a subspace training implementation, with lines ($m=2$) and triangles ($m=3$) of full corner weights. The teacher $\bar p$ costs $m$ extra forward passes per step, with no backward passes through it.

Beyond accuracy, the program has three analyses. First, we will track corner spread, prediction disagreement among corners, and the loss barrier between corners throughout training, to confirm empirically that the spread-invariance argument holds under the actual optimizer. Second, we will test whether the corners' averaged prediction $\bar p$ is actually better than $\theta_c$ during training. If the center already matches its own corners, the teacher has nothing to give, and the idea fails cleanly. Third, we will try an optional online version of LatentBE's diversifying perturbation. It would craft inputs where the corners disagree but the center is confident, using the corners themselves in place of a pre-trained ensemble. We report it separately so it cannot carry the main result.

Baselines

The naive reference is a single network trained normally. The CIFAR-100 ResNet-32 top-1 error is reported as 28.39% in the multi-branch diversity paper and 28.76% in OKDDip. We rerun it under our schedule.

The ablation that isolates the key ingredient is plain subspace training with midpoint evaluation, using identical corners, schedule, and cosine penalty, but no distillation term. We found no reported CIFAR-100 or ImageNet number for this method on Lacuna. We therefore rerun it, giving it the extra GPU-hours our teacher passes consume, spent as longer training. A second ablation distills the corner consensus into each corner rather than into the center. The theory predicts that this version collapses the spread; the center version should not. Beating both ablations at matched compute is what the core claim requires.

The strongest online-distillation competitor is multi-branch diversity enhancement. It reports 24.84% top-1 error on CIFAR-100 with ResNet-32, against 25.40% for OKDDip. We rerun it on the same backbone, augmentation, and epoch budget, with a hyperparameter search of the same size as ours. Beating it means lower error with a deployed model of identical size and FLOPs, averaged over at least three seeds.

On ImageNet, the reference points are collaborative learning's ResNet-50 at 22.29% top-1 error versus a 23.47% baseline, and KDCL's ResNet-50 at 78.2% top-1.

The offline reference is LatentBE. It reports 80.02% CIFAR-100 accuracy, NLL 0.792, and 0.107 ms per image. The file does not give the backbone for these numbers, so we rerun LatentBE on our backbone and charge it the full cost of training its teacher ensemble. Matching its accuracy and NLL within seed noise at substantially lower total GPU-hours counts as a win.

A deep ensemble serves as an upper reference for calibration and out-of-distribution detection. On CIFAR-10 it reaches 95.2% accuracy, CIFAR-10C expected calibration error of about 0.206, and SVHN AUROC of 0.924; a single model reaches 94.1%, about 0.323, and 0.893 respectively.

Why not just…?

Why not just use SWA? SWA is one network with almost no overhead, and on CIFAR-100 it improves ResNet-164 by 1.3% over SGD. It averages points along a single trajectory, however, and supplies no teacher signal. The subspace paper argues that such points explore less of weight space than a trained region does. We include SWA, and SWA applied on top of our center model, at matched epochs. If plain SWA ties our method, the distillation term is not earning its cost, and we will say so.

Why not just train the subspace and take its midpoint? This is the real threat. The subspace paper already shows that the midpoint gets flat-minimum benefits in one run at zero inference cost. Our entire contribution is the difference between that midpoint and a midpoint that has also been distilled from its corners. The compute-matched ablation above settles it directly. Our analysis of whether $\bar p$ beats $\theta_c$ during training predicts the answer in advance: if the corners' ensemble is never better than the center, we expect no gain.

Why not keep collaborative learning and average the heads at the end? Heads trained toward a shared consensus homogenize, as the multi-branch diversity paper and OKDDip document. Averaging near-identical heads therefore gives back roughly one head. Heads that are not homogenized have no guarantee of lying in a shared low-loss region, so averaging them may cross a loss barrier. We run post-hoc head averaging on collaborative learning as an extra baseline and report its loss barrier alongside its accuracy.

Why not just run LatentBE? LatentBE works, but it needs a deep ensemble trained first, plus repeated passes through that ensemble during distillation. That is exactly the sequential cost this proposal removes. The comparison is framed in total GPU-hours. If LatentBE wins clearly on NLL or out-of-distribution detection even after accounting for its cost, we will conclude that a pre-trained teacher carries information online corners cannot supply.

Proposed Experiments

We run CIFAR-100 with ResNet-32 for direct comparison with the online-distillation numbers above. We run CIFAR-10 with CIFAR-10C and SVHN for calibration and out-of-distribution detection against the efficient-ensemble study. ImageNet-1K with ResNet-50 tests scale. PACS and OfficeHome test domain shift, against Weight Averaging Improves Knowledge Distillation under Domain Shift, which reports 86.6% on PACS with ResNet-18 but uses an offline teacher. Metrics are top-1 accuracy, NLL, expected calibration error, out-of-distribution AUROC, peak training memory, training GPU-hours, and inference latency.

Three checks are decisive. The first is the spread test: corner spread and disagreement should hold steady under center distillation and collapse under corner distillation. The second is the compute-matched gain of center distillation over the plain subspace midpoint, across seeds. The third is the total-compute comparison against LatentBE.

If the first check passes but the second fails, the geometric argument is right but the online teacher is too weak. That would point future work toward stronger diversity sources, such as the online perturbation variant, rather than toward the distillation loss itself.