Supported Margins: Separating Trustworthy from Borrowed Confidence for Selective Classification Under Shift

A classifier that may abstain ("selective prediction") is useful only if it abstains on the inputs it would get wrong. The usual signal is how decisively the model separates its top answer from the runner-up, called the margin. Margins fail under distribution shift for a specific reason. A corrupted or shifted input can push the network's internal features in directions that the training data never varied along. When those unfamiliar directions happen to line up with the classifier's weights, they inflate or deflate the margin, and the model looks confident on evidence it has never been calibrated on. We propose to measure this directly. Singular value decomposition (SVD), which factors a matrix into ranked orthogonal directions of variation, finds the subspace that training features actually occupy. Because the final layer is linear, the margin splits exactly into a part produced by features inside that subspace and a part produced by the residual outside it. We call the first part the "supported" margin and the second the "borrowed" margin. The proposed selection rule accepts a prediction only when its supported margin is large and its borrowed margin is small.

Two existing results frame the opportunity. ViM uses the size of the SVD residual as an out-of-distribution score. Selective Classification Under Distribution Shifts then shows that such out-of-distribution scores, ViM included, transfer poorly to selective classification, while plain margin scores do well. Our reading is that both observations hold for one shared reason. Residual size flags every off-subspace input, including benign shifts that leave the decision unchanged. The raw margin ignores where its evidence comes from. Projecting the residual onto the decision direction keeps what each signal gets right and drops what each gets wrong.

Core Claim

Let $h(x)\in\mathbb{R}^D$ be the penultimate feature and $w_c, b_c$ the final-layer weights and bias for class $c$. Let $\mu$ and $V_k\in\mathbb{R}^{D\times k}$ be the mean and the top-$k$ right singular vectors of centered in-distribution training features. Write $x_\perp=(I-V_kV_k^\top)(h(x)-\mu)$ for the off-subspace residual. For the predicted class $c$ and runner-up $c'$, the logit margin decomposes exactly:

$$m(x) = \underbrace{(w_c-w_{c'})^\top\big(\mu + V_kV_k^\top(h(x)-\mu)\big) + b_c-b_{c'}}_{m_\parallel(x)\ \text{(supported)}} \;+\; \underbrace{(w_c-w_{c'})^\top x_\perp}_{m_\perp(x)\ \text{(borrowed)}}.$$

The proposed confidence score is $s(x)= \big(m_\parallel(x) - \lambda\,|m_\perp(x)|\big)/\|w_c-w_{c'}\|$, with $\lambda\ge 0$. Dividing by the weight-difference norm makes the score a geometric distance, in the spirit of the scale-invariant margins of Selective Classification Under Distribution Shifts. Two special cases matter. Setting $\lambda=0$ gives a margin computed on SVD-filtered features. Replacing $-\lambda|m_\perp|$ with $+m_\perp$ recovers that paper's raw geometric margin, RLgeo-M. If $m_\parallel$ is negative, the in-subspace evidence prefers another class, and that disagreement is itself grounds for rejection.

The claim is falsifiable on three counts. First, on in-distribution data $s$ should match raw margins. Second, under corruption-type covariate shift it should beat both raw margins and ViM-style residual scores on risk-coverage metrics. Third, the gain should come from aligning the residual with the decision direction, not from the residual's size.

Research Program

The method is post-hoc. It needs one SVD of training features, here computed with a randomized solver as in rsvd, and one $D\times k$ projection per input. Training is unchanged.

The first study is descriptive. For pretrained ImageNet models, we measure how $|m_\perp|$ is distributed for correct versus incorrect predictions, comparing clean data, ImageNet-C corruptions and OpenImage-O. The hypothesis predicts that misclassified shifted inputs carry disproportionately large borrowed margins. Large-residual inputs that are still correct should show small $|m_\perp|$, because their residual is orthogonal to the decision direction. This is the mechanism that would explain why ViM's residual norm over-rejects correct samples.

The second study selects the rank $k$ and the weight $\lambda$ on clean validation data only, so that no shifted data leaks into tuning. A class-conditional variant computes the subspace from features of classes $c$ and $c'$ only. This tests the class-geometry question raised by Selective Classification via One-Sided Prediction, which builds per-class decision regions without modeling their spectra.

The third study concerns guarantees. We set thresholds with the Selection with Guaranteed Risk procedure of Selective Classification for Deep Neural Networks. That procedure assumes test data comes from the same distribution as validation data, so the question is how often its target risk is violated on shifted test sets. Our hypothesis is that thresholds on $s$ transfer better than thresholds on softmax or raw margins, because $s$ discounts exactly the evidence the validation set could not calibrate.

Baselines

The naive reference is Softmax Response, the maximum predicted probability. Selective Classification for Deep Neural Networks uses it to report 1% error at 78% coverage on CIFAR-10.

A training-side competitor is Self-Adaptive Training with entropy minimization followed by Softmax Response, from Towards Better Selective Classification. It outperformed SelectiveNet and reported up to 80% relative error reduction on ImageNet subsets. We train it on CIFAR-10 with the same backbone and epoch budget as our base model. We also apply our score on top of it, which tests whether the two are complementary.

The strongest competitor under shift is the pair of margin scores RLgeo-M and RLconf-M from Selective Classification Under Distribution Shifts. They are the best overall when all shift types are present on ImageNet with EVA. The file quotes no scalar for them (no reported result found on Lacuna for RLgeo-M AURC on ImageNet-C or iWildCam), so we rerun them on identical backbones and splits. Beating them means a lower area under the risk-coverage curve (AURC) on the combined shifted pool and higher coverage at a 2% risk target. It must also hold across both models and at least two of the three shift families, and the clean-data AURC must stay within noise.

The geometric competitor is ViM, which reports 90.91% average out-of-distribution AUROC with BiT-S on ImageNet-1K. That is an out-of-distribution metric, not a selective one. We evaluate ViM, and Mahalanobis distance, as selection functions.

On CIFAR-10 we include One-Sided Prediction, which reports 64.1% coverage at 0.5% error against 59.3% for Softmax Response Selective Classification via One-Sided Prediction. Its 64.1% is the bar for the strict-error regime.

Three ablations isolate the key ingredient:

  • replacing $|m_\perp|$ with the residual norm $\|x_\perp\|$, which is a ViM-like penalty that ignores the decision direction;
  • replacing $V_k$ with a random $k$-dimensional subspace;
  • a validation-tuned linear fusion of RLgeo-M and the ViM score.

All methods share frozen backbones, the same clean validation split, and the same hyperparameter grid size.

Why not just…?

Why not just use the geometric margin alone? This is the most serious threat. The evidence that margins already beat out-of-distribution scores comes from Selective Classification Under Distribution Shifts, and if the borrowed margin is tiny in practice, $s$ collapses to RLgeo-M. The descriptive study settles this directly. If $|m_\perp|$ does not differ between shifted errors and shifted correct predictions, the idea fails, and we would report it as a negative result.

Why not just combine margin with ViM's residual score? Score fusion is cheap and plausible. It cannot, however, tell a large residual that moves the decision from one that does not, and it is that indiscriminate penalty on benign shift that makes residual scores mis-rank correct samples. The fusion baseline and the norm-penalty ablation test this. If fusion matches $s$, the decision-aligned decomposition adds nothing.

Why not just train a better-regularized classifier? Entropy-minimized training Towards Better Selective Classification improves softmax ranking. Its authors note, though, that its thresholds assume validation and test share a distribution. Our method needs no retraining and can sit on top of such a classifier. The stacked condition shows whether the gains are additive.

Why not MC-dropout? It requires many forward passes. Single-pass scores already match or beat it SelectiveNet, so we report it only on CIFAR-10 for completeness.

Proposed Experiments

On ImageNet-1K with EVA and BiT-S backbones, we evaluate on clean validation data, ImageNet-C, OpenImage-O, iNaturalist, and a mixed pool combining in-distribution errors, label shift and covariate shift, following the generalized selective classification protocol. Natural shift uses iWildCam and Amazon. CIFAR-10 and CIFAR-10-C cover the strict-error regime.

Metrics are AURC, selective risk at 80% and 95% coverage, and coverage at target risks of 0.5% and 2%. We also report the rate of Selection with Guaranteed Risk violations on shifted sets at a target risk $r^*$ and confidence $\delta=0.05$.

The decisive checks are three. First, $s$ must beat RLgeo-M on corruption shift without losing on clean data. Second, the random-subspace and residual-norm ablations must each erase most of the gain. Third, guarantee violation rates must fall relative to raw margins.

We predict in advance that iNaturalist will show little gain. ViM's authors note that near-semantic out-of-distribution inputs there have small residuals, so both $m_\perp$ and $\|x_\perp\|$ will be small. A gain there would indicate the method works for reasons other than the one proposed.