Ranking Architectures for Single-Pass Distance-Aware Uncertainty Before Training

A classifier should become less confident as an input moves farther from anything it saw during training. Most networks do not behave this way. One family of methods builds this behavior into a single model that needs only one forward pass, with no ensemble and no repeated sampling. These are called distance-aware uncertainty methods. They have two parts. The first is a constraint that stops the feature extractor from mapping very different inputs to the same point, a failure called feature collapse. The second is an output head that turns distance in feature space into uncertainty. Every such method so far was bolted onto a hand-designed backbone such as a ResNet, Wide ResNet, or BERT. Nobody knows whether the backbone's wiring matters for this kind of uncertainty once accuracy is accounted for. If it does matter, nobody knows whether good wirings can be found without training thousands of candidates.

This proposal answers both questions at once. We first measure how much of the variation in out-of-distribution (OOD) detection across a standard architecture benchmark is explained by architecture beyond what accuracy already explains. We then test whether a cheap measurement of distance preservation, taken at initialization or after one epoch, can find the architectures that do unusually well. The first result tells the field whether searching for uncertainty-friendly architectures is worth doing at all. The second tells it whether such a search can cost minutes rather than thousands of GPU hours.

Core Claim

We claim that, under a spectral-norm constraint, a cell's topology largely determines whether the resulting network can preserve distances. Topology here means which edges are identity paths, which are convolutions, and which are pooling or zero. We further claim that a cheap empirical distortion statistic will rank architectures by their distance-aware OOD performance after controlling for accuracy, and that existing accuracy-oriented proxies will not.

The statistic works as follows. We pass pairs of training images, including pairs built from perturbations at several scales, through a spectrally normalized candidate network. For each pair we compute the ratio between feature-space distance and input-space distance:

$$r_{ij} = \frac{\|f(x_i) - f(x_j)\|}{\|x_i - x_j\|}, \qquad D(f) = \frac{q_{0.05}(r)}{q_{0.95}(r)}$$

Here $f$ is the feature extractor, $x_i, x_j$ are inputs, and $q_p$ is the $p$-th quantile over pairs. $D$ is therefore an empirical ratio of the network's lower and upper Lipschitz constants. A value near 1 means the network neither squashes nor stretches distances. A value near 0 means some distinct inputs are nearly merged.

The statistic uses only training-distribution data, so no OOD set is needed at search time. That matters because tuning on OOD data is one of the known weak points of these methods.

One property of $D$ must be faced directly, because it shapes the whole design. $D$ alone rewards networks that are almost the identity map. These are the parameter-free, all-skip-connection cells that NAS-Bench-201 shows differentiable search falls into. The claim is therefore never that $D$ should be maximized. The claim is that $D$ predicts uncertainty quality at a given accuracy. The proxy is used jointly with an accuracy proxy to trace a two-objective frontier.

Where This Sits in Prior Work

The single-pass methods supply the machinery we reuse. SNGP applies spectral normalization to residual layers and replaces the final layer with a random-feature Gaussian process. DUE shows that unconstrained deep kernel learning actively encourages collapse. It enforces a two-sided (bi-Lipschitz) constraint through residual connections plus spectral normalization and uses an inducing-point Gaussian process head. DUQ instead uses a two-sided gradient penalty and class centroids. All three fix the backbone by hand. We hold their heads and constraints fixed and vary the backbone.

DUE's argument is also the reason we expect topology to matter. Its guarantee leans on an identity path plus a contracting residual branch, and that is a property of the cell's wiring.

Two studies motivate the question and also constrain it. Training, Architecture, and Prior for Deterministic Uncertainty Methods finds that backbone expressiveness is the main driver of uncertainty quality. It also finds that bi-Lipschitz constraints help OOD detection but hurt generalization under natural shift, and that the encoder and head need separate learning rates. Its first finding is a real threat to us: it suggests that bigger and more accurate backbones may simply be better, which would leave nothing for topology search to find. Its second finding is why we report a frontier rather than a single score. Its third finding we adopt directly in all training runs. NADS shows that operation choices such as max pooling hurt anomaly detection. That is evidence architecture matters, but NADS works with flow-based generative models and multi-model ensembles, not single-pass classifiers.

The uncertainty-aware search work is the closest, and each piece differs in a specific way:

  • μDARTS adds Concrete Dropout and a variance penalty to differentiable search. Its uncertainty comes from multiple stochastic passes, which van Amersfoort et al. show can stay overconfident far from the data.
  • Neural Ensemble Search searches over heterogeneous ensembles, which multiplies deployment cost.
  • FNAS uses uncertainty only in its predictor of rewards, not in the searched models.
  • NAS-OoD optimizes accuracy under shift and produces no uncertainty estimate.

None of these searches for a single deterministic backbone paired with a distance-aware head.

The cheap-evaluation work supplies our method and our competitors:

  • Zero-Cost Proxies for Lightweight NAS scores untrained networks with pruning-style signals such as Synflow, SNIP, and jacob_cov.
  • EZNAS evolves such scores automatically.
  • Dextr uses singular-value conditioning of feature maps plus output curvature. It is the nearest relative of our statistic, but it targets trainability and accuracy, and it measures channel collinearity rather than how pairwise distances between inputs are preserved.
  • CRoZe and the adversarial robustness proxy show that proxies can target properties other than accuracy. Both measure stability against small perturbations near the data, which is roughly the opposite of what we need: sensitivity to inputs far from it.

Research Program

Ground truth. The first deliverable is a ground-truth table that does not yet exist. It covers a stratified sample of roughly one thousand architectures from the NAS-Bench-201 cell space. The sample is stratified by benchmark accuracy and by number of identity edges, so that the degenerate skip-heavy region and the top decile are both well covered.

Each architecture is retrained on CIFAR-10 with spectral normalization in every convolution and with two heads in separate runs: the SNGP random-feature Gaussian process head and the DUE inducing-point head. Training uses decoupled encoder and head learning rates, as Charpentier et al. require. We use three seeds per architecture so that seed noise can be estimated and compared against architecture effects.

For every trained model we record clean accuracy, calibration error, and AUROC for OOD detection against SVHN (far OOD) and CIFAR-100 (near OOD). We also record accuracy on corrupted CIFAR-10, which serves as the generalization-under-shift axis in the trade-off Charpentier et al. describe. A smaller replication on CIFAR-100 and ImageNet-16-120 checks that the findings are not specific to one dataset. We will release the table, publish its compute cost, and treat it as a contribution in its own right, the uncertainty counterpart to NAS-Bench-201's accuracy table.

Variance analysis. The second deliverable answers whether there is anything to search for. We regress OOD AUROC on clean accuracy and parameter count across architectures. We then compare the variance left over between architectures against the variance between seeds of the same architecture. If architecture explains little beyond accuracy and size, the honest conclusion is that accuracy-driven search plus a post-hoc distance-aware head is sufficient, and we report that.

Proxy evaluation. The third deliverable tests $D$ at three costs: at initialization, after 100 optimizer steps, and after one epoch. We test it with and without spectral normalization applied during measurement. Spectral normalization is the ingredient we hypothesize makes the statistic informative, because without it the upper Lipschitz constant of a random network is uncontrolled. The success criterion is rank correlation with the leftover AUROC after accuracy is regressed out. Correlation with raw AUROC is reported too, but it can be inflated by accuracy alone. The step-limited variants exist because DUE shows collapse is driven by training. A purely initialization-time statistic may miss it, and we want to know at which point cost and fidelity trade off.

Baselines

Naive reference. We take the highest-accuracy NAS-Bench-201 architecture and give it the same spectral normalization and the same head. This is exactly the skeptic's recipe: search for accuracy, then add distance awareness afterward. A random architecture of matched parameter count gives a second, weaker floor.

To beat the naive reference, the proxy-selected architecture must reach a higher SVHN AUROC than the accuracy-selected one at clean accuracy within one point. The gap must exceed the seed spread, averaged over the three seeds. Both architectures get identical training recipes and identical head tuning budgets, with head hyperparameters tuned on in-distribution validation data only.

Strongest existing cheap selector. This is the zero-cost vote ensemble of Synflow, jacob_cov, and SNIP from Zero-Cost Proxies for Lightweight NAS. It is reported to reach a Spearman correlation of 0.82 with accuracy on NAS-Bench-201. We add EZNAS-A, reported at Kendall Tau 0.65, and Dextr. We found no reported result on Lacuna for any zero-cost proxy's correlation with OOD AUROC or with distance-aware uncertainty, so we compute these ourselves on the same thousand architectures and the same minibatches as $D$.

Beating these selectors means two things. First, $D$ must show a higher rank correlation with accuracy-adjusted AUROC. Second, combining $D$ with the vote ensemble must give a two-objective frontier that dominates the vote ensemble alone in its top decile, which is where existing proxies are known to fail.

Strongest uncertainty method for absolute reference. These are the hand-designed single-pass models and the ensembles. DUE is reported at an AUROC of 0.958 on CIFAR-10 versus SVHN. In the same comparison, DUE reports SNGP at 0.940 and a 5-model Deep Ensemble at 0.967, all with Wide ResNet backbones. NAS-Bench-201 cells are much smaller, so we retrain a Wide ResNet DUE under our pipeline rather than comparing against the published number. We also evaluate an architecture transferred from our search at matched parameter count.

μDARTS and a five-member NES ensemble drawn from the same space serve as the multi-pass competitors. We report latency next to AUROC so the single-pass cost advantage is visible.

Ablations. Three ablations isolate the key ingredient: $D$ measured without spectral normalization, $D$ restricted to the upper quantile only (an ordinary smoothness measure), and parameter count alone.

Why not just…?

Why not just put an SNGP or DUE head on the best-accuracy architecture? This is the main threat, and it may win. Charpentier et al. found expressiveness to be the dominant factor, which points toward this outcome. DUE's analysis cuts the other way: collapse depends on how the backbone is constrained, and constraints act differently on different cell wirings. SNGP's authors also note that convolutional layers need looser norm bounds to keep accuracy. Rather than argue, the variance analysis settles it. If accuracy-adjusted AUROC varies across architectures no more than across seeds, this recipe is the answer and the search agenda should stop there.

Why not just use existing zero-cost proxies, on the assumption that accuracy brings good uncertainty? jacob_cov measures how correlated activations are across inputs within a batch. That is related to collapse, so it is a serious competitor, and it is included under identical conditions. The structural reason to expect a gap is that accuracy proxies reward contraction toward class-discriminative directions. Distance-aware heads need the opposite at the lower end of the distance spectrum. The rank correlation with accuracy-adjusted AUROC is the deciding measurement.

Why not just train each candidate briefly and measure OOD AUROC directly? Budgeted training of this kind, as in Rethinking Performance Estimation in Neural Architecture Search, is the natural multi-fidelity alternative. It needs an OOD set during search, and that set would then leak into the selection. It may still outperform an initialization-time statistic. For that reason one-epoch AUROC, computed on a held-out OOD set that is never used for final evaluation, is added as an oracle-style upper reference. Our one-epoch $D$ is compared against it at equal cost.

Why not just use an ensemble or Monte Carlo dropout? SNGP reports 6.25 ms per example against 38.10 ms for a 10-model Deep Ensemble on Wide ResNet. Van Amersfoort et al. also show that ensembles stay confident in empty regions far from the data. The comparison is still run, because a five-member NES ensemble of small cells might beat a single model on AUROC at a latency some users would accept.

Decisive Checks

The project turns on three numbers.

The first is the ratio of between-architecture to between-seed variance in accuracy-adjusted OOD AUROC. If this ratio is near one, architecture search for single-pass uncertainty is not worth doing, and that negative result is publishable on its own.

The second is the rank correlation of spectrally normalized $D$ with accuracy-adjusted AUROC, compared with the zero-cost vote ensemble and with the version of $D$ measured without spectral normalization. If spectral normalization does not raise the correlation, our structural explanation is wrong.

The third is whether the frontier selected by $D$ plus the vote ensemble yields, on held-out CIFAR-100 and ImageNet-16-120 runs, an architecture that beats the accuracy-selected architecture plus head on SVHN AUROC at matched clean accuracy, without losing more corrupted-CIFAR-10 accuracy than it gains in detection. If it does, the detection-versus-generalization trade-off Charpentier et al. report is partly an artifact of using fixed backbones, and it can be eased by search costing minutes.