Which Student Can Hold a Teacher's Doubt? Training-Free Search for Uncertainty Distillation

A deep ensemble, meaning several independently trained networks whose predictions are averaged, is one of the most reliable ways to get honest uncertainty from a neural network. It is also several times more expensive to run than a single model. Uncertainty distillation compresses the ensemble into one "student" network. The student learns to reproduce the average prediction and also how much the ensemble members disagree, and it does this in a single forward pass. Work in this area almost always fixes the student's architecture in advance, for example a ResNet-18. Architecture search could choose a better student, but ranking candidates by actually distilling into each one costs hundreds of GPU-hours. The cheap scoring rules that architecture search uses to avoid training, called zero-cost proxies, were all built to predict accuracy.

This note proposes to measure, and then exploit, one possible fact: the architecture that best absorbs a teacher's disagreement may differ from the one that best absorbs its mean prediction. We would build the first benchmark of uncertainty-distillation outcomes over a tabular architecture space. On it we would test a new training-free score. The score checks whether a randomly initialized student's function space lines up with the directions in which the teacher's members disagree. If uncertainty distillability turns out to be predictable from accuracy alone, the benchmark will show that clearly, and that is a publishable negative result that saves the field a search problem.

Core Claim

Our hypothesis is that, across the 15,625 cells of NAS-Bench-201, the ranking of architectures by distilled uncertainty quality differs substantially from their ranking by distilled accuracy. Uncertainty quality here means out-of-distribution detection and the error in matching the teacher's disagreement. We further hypothesize that a kernel-alignment score computed at initialization, from one mini-batch, predicts the uncertainty ranking better than accuracy-oriented proxies at comparable cost.

The score works as follows. Let $p_m(x)$ be the softmax output of teacher member $m$ out of $M$, and let $\bar p(x)$ be their mean. The teacher disagreement kernel on a batch is

$$K_T(x,x') = \frac{1}{M}\sum_{m=1}^{M} \big(p_m(x)-\bar p(x)\big)^\top \big(p_m(x')-\bar p(x')\big).$$

This kernel is large for pairs of inputs on which the ensemble members disagree in the same way. Let $K_S(x,x')$ be the inner product of the student's output Jacobians with respect to its weights at random initialization, known as the empirical neural tangent kernel. It describes which input pairs the untrained student will move together when it learns. The proxy is the centered kernel alignment between $K_S$ and $K_T$.

The intuition comes from linearized training. A student whose tangent kernel already couples the inputs that the teacher finds jointly ambiguous can fit the teacher's spread with small weight changes. A student whose kernel only couples inputs by class can fit the mean but will smear the spread.

Relation to Prior Work

The closest work is DisWOT. It searches for distillation students without training them, scoring each uninitialized student by how well its Grad-CAM maps and inter-sample feature similarities match the teacher's. It reports a 1.62% top-1 gain over standard knowledge distillation for ResNet-50 to ResNet-18 on ImageNet. That result supplies our premise that the best standalone architecture need not be the best student. However, DisWOT's scores compare deterministic features of a single teacher, and nothing in them sees disagreement between ensemble members. Our score is built only from the disagreement component of the teacher. Our main ablation swaps it for the kernel of the teacher mean, which isolates exactly that difference.

The accuracy-proxy literature constrains the design. EZNAS evolves scoring formulas that reach Kendall's τ of 0.65 against NAS-Bench-201 CIFAR-10 accuracy, compared with 0.57 for Synflow and NASWOT. NAS-Bench-Suite-Zero shows that most raw proxies mainly track parameter count and that combining 13 of them in a boosted-tree predictor raises Spearman correlation by up to 42%. Both findings warn us that any new score must beat parameter count and must be tested within capacity strata.

Three uncertainty-aware search methods motivate the gap but do not fill it. μDARTS searches with a variance penalty but deploys Monte Carlo dropout, which needs multiple passes. NADS learns a distribution over architectures and deploys an ensemble of sampled architectures. It is nonetheless evidence that architecture choice matters for detecting out-of-distribution inputs. Neural Ensemble Search shows that architecturally diverse ensembles cut calibration error by up to 40%, but it trains and deploys many full models. We reuse its ensembles as one of our teachers.

Two distillation papers supply the student's training method. Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage? shows that evidential networks trained alone produce epistemic uncertainty that does not shrink with more data. It also shows that grounding the network through bootstrap distillation raises CIFAR-100 versus SVHN detection AUROC from about 64.7 to about 85.6. We therefore train students with a Dirichlet output head distilled from ensemble members, not with standalone evidential losses. GPED found that the teacher–student gap peaks at intermediate task difficulty and searched only through channel pruning during full training. Our benchmark tests whether its "capacity must match" finding extends to cell topology and whether the effect can be scored before training.

Research Program

The first step is building the benchmark. For CIFAR-10 and CIFAR-100 we train two teachers. One is a 5-member deep ensemble of a fixed ResNet. The other is an architecturally diverse ensemble selected by NES-style search within NAS-Bench-201 itself. We then sample about 1,000 NAS-Bench-201 cells, stratified by parameter count and by tabulated accuracy decile. Each is distilled under a fixed recipe into a Dirichlet student. The teacher trains on the student's concentration parameters by matching member predictions, following the bootstrap-distillation setup.

For each student we record accuracy, negative log-likelihood, expected calibration error, AUROC for CIFAR versus SVHN, and the error in matching the teacher's mutual information on held-out and corrupted inputs. We subtract the same student architecture's plain-KD numbers to obtain a "disagreement gain." A subset of 150 cells also goes on ImageNet-16-120 to test transfer. At an estimated one to two GPU-hours per small-cell distillation, the program costs roughly 1,500–2,500 GPU-hours, and we would release it as a lookup table.

The second step is analysis. We ask how far the uncertainty ranking departs from the accuracy ranking. We measure this departure both globally and within the top 10% of cells, which is where EZNAS notes proxies struggle, and within parameter strata. We also ask which operations predict it. NAS-Bench-201 documents skip-connection-dominated cells as a failure mode, and NADS observed that pooling hurts detection. The third step evaluates the alignment proxy and, if it holds up, runs a proxy-guided search whose selected student is distilled at full budget.

Baselines

The naive reference is the student a practitioner would pick without thinking: the highest-accuracy NAS-Bench-201 cell under the parameter budget, distilled with the same recipe. Its tabulated CIFAR-10 accuracy is 94.37%. A second reference is parameter count used directly as the ranking score, since NAS-Bench-Suite-Zero shows that many proxies reduce to it.

The accuracy-proxy baselines are Synflow, NASWOT, and EZNAS-A. On accuracy their reported Kendall's τ is 0.57, 0.57, and 0.65, respectively (EZNAS). We found no reported result on Lacuna for any zero-cost proxy's correlation with distilled uncertainty quality, so our benchmark supplies the first such numbers.

The strongest existing search competitor is DisWOT's combined semantic and relation score, reimplemented against the ensemble-mean teacher. Its only reported figure is the 1.62% top-1 gain over KD on ImageNet. It has no uncertainty result. We also include the 13-proxy boosted-tree surrogate, trained on the same number of labeled benchmark cells as any learned variant of ours.

The key ablation replaces $K_T$ with the kernel of the teacher mean $\bar p$. If that version does as well, the disagreement ingredient contributes nothing.

All scores use the same batch size and seeds. Each gets a comparable wall-clock budget, measured in forward and backward passes. They are evaluated on the same held-out cells. Beating a baseline means a higher Kendall τ with disagreement gain and out-of-distribution AUROC, significant over five sampling seeds, and holding within parameter strata. For search, the proxy-selected student must beat the naive student's detection AUROC at equal parameters and single-pass latency. As an external anchor, LatentBE reports CIFAR-100 NLL of 0.792 against 1.038 for standard KD on a hand-designed backbone.

Why not just…?

Why not just distill into the most accurate architecture? If uncertainty rankings track accuracy rankings, this is the right answer. The first analysis is designed to show whether that is so, and we will say so if it is. The contrary evidence is indirect: DisWOT shows that student choice matters for accuracy, and NADS and GPED suggest that architecture shapes out-of-distribution behavior and the size of the distillation gap. None of these studies measured the specific comparison we need.

Why not just distill each candidate for one to five epochs as a low-fidelity estimate? This is the most serious threat. Short training is the standard multi-fidelity move, and Rethinking Performance Estimation in Neural Architecture Search formalizes choosing such budgets. We will measure rank correlation per GPU-second for 1-, 3-, and 10-epoch distillation alongside the proxy. If a short budget dominates at similar cost, the contribution becomes the benchmark and the fidelity recommendation, not the proxy.

Why not just use temperature scaling or Monte Carlo dropout on a searched model? Temperature scaling rescales confidence and cannot reorder which inputs look unfamiliar, so we report it only as a calibration reference. Monte Carlo dropout requires multiple passes, which reintroduces the latency that distillation is meant to remove. The four-pass versus single-pass latency ratio in LatentBE, 0.390 against 0.107 ms per image, shows the size of that cost.

Decisive Checks

Three results settle the project. First, if Kendall τ between the distilled-accuracy and disagreement-gain rankings exceeds about 0.9 globally and within the top decile, we stop at the benchmark and report the negative result. Second, the alignment proxy must beat both parameter count and its mean-kernel ablation in partial correlation controlling for size. If it does not, the proxy claim fails. Third, the ranking must transfer: a proxy fitted and validated on CIFAR-100 must retain its advantage on ImageNet-16-120 and under the architecturally diverse teacher. If it does not, the score is reading a quirk of one teacher rather than a property of the student.