A classifier that is allowed to say "I don't know" is only as good as its ranking of which inputs to refuse. The most reliable way to get that ranking is an ensemble, meaning several independently trained models whose predictions are averaged. Averaging lowers confidence exactly where the models disagree, so those inputs get refused first. Ensembles are expensive, though, and modern ensembles are increasingly redundant: their members make the same mistakes. Kernel machines are classifiers that predict by comparing a new input against a stored set of training examples, the "support vectors." They have a long history of being compressed by throwing most of those examples away. The usual compression rules discard the examples whose labels look uncertain or noisy. Those are exactly the examples a refusal mechanism needs.
We propose to attach a small kernel machine to a large network's frozen internal features and use it as a cheap second opinion for selective classification. The machine is compressed with a pruning rule that deliberately keeps the training points on which an ensemble of sub-machines disagrees. The two-sentence version: a kernel classifier on a network's features is a structurally different predictor that costs well under 1% of a forward pass if it keeps only a few thousand training points. We claim that which points it keeps matters more for abstention than for accuracy, so pruning must be chosen to preserve ambiguity, not just accuracy.
Core Claim
The proposal makes three linked claims. First, accuracy-oriented sparsification of kernel machines degrades selective risk much more than it degrades accuracy. Cross-Training is an example: it trains SVMs on data subsets and discards points by their margin statistics (Breaking SVM Complexity with Cross-Training). L0-reweighted least-squares SVMs are another (Sparse LS-SVMs with L0 - norm minimization). Both kinds of pruning flatten the region where confidence should drop. The research file flags this tension (Gap 1), but no one has measured it.
Second, the same ensemble of sub-machines that Cross-Training uses for cleaning can be used the other way round. Points whose margin varies across sub-machines are the training-set analogue of the "ambiguous samples" that Top-Ambiguity Samples Matter identifies as the whole source of ensemble gains in selective classification. Retaining them should keep abstention quality at a small support-vector budget.
Third, the resulting sparse kernel head combined with the base network forms an asymmetric pair in the sense of Asymmetric Duos. That paper pairs a large network with a tiny architecturally distinct network and gets deep-ensemble-like selective coverage for 10–20% extra FLOPs. Our sidekick is cheaper still, and its source of diversity is different. It shares the base's features but replaces the base's linear output layer with a local, example-based decision rule.
The third claim is the riskiest. A sidekick built on the base model's own features may simply inherit the base model's mistakes. This is the redundancy problem documented for shared-checkpoint models (Neyshabur et al.; Sarna et al.) and for model families (Hidden Clones). We treat that as an empirical question with a direct measurement, not an assumption.
Research Program
The base model $f$ is a standard image classifier trained on most of the training set. A held-out split of 10–20% is reserved, because a kernel machine fit on features the network has already memorized would see nearly perfectly separated data and learn nothing about the network's errors. Every baseline that is fit after the base model gets the same held-out split. We extract penultimate features $z = \phi(x)$ on the held-out split. We then run Cross-Training's first stage: split the held-out features into $s$ disjoint parts, train an RBF-kernel SVM on each, and score every held-out point $i$ under all $s$ machines. This yields the mean $\mu_i$ and standard deviation $\sigma_i$ of its signed margin.
Cross-Training keeps a band of near-boundary points and discards consistent violators and deep interior points. Instead, we rank points by a retention score that favors disagreement and low margin:
$$r_i = \frac{\sigma_i + \lambda\,\mathbb{1}[f \text{ errs on } x_i]}{|\mu_i| + \epsilon},$$
Here $\lambda \ge 0$ adds credit for held-out points that the base network itself misclassifies, and $\epsilon$ is a small constant. Points are retained per class up to a total budget $B$, which directly addresses the class-imbalance failure mode of threshold-based pruning noted for Cross-Training. A small random sample of interior points is added so the machine still knows where the confident regions are. A final kernel machine is trained on the retained set, using either an SVM or a least-squares SVM so we can compare against ISLS-SVM directly.
At test time the sidekick's class scores are temperature-scaled and added to the base logits with weights tuned on validation data. This is exactly the Duo combination rule. Using the same rule makes any difference attributable to the sidekick rather than to the combiner. The confidence used for abstention is the maximum combined softmax probability.
The cost argument is explicit. With $B = 2{,}000$ retained points in a 2,048-dimensional feature space, the sidekick costs about four million multiply-adds per input. That is roughly 0.1% of a ResNet-50 forward pass, against the 10–20% that a sidekick network adds.
The analysis has three parts. First, we trace how AURC and accuracy change as $B$ shrinks under each pruning rule. AURC is the area under the curve of error on accepted inputs versus fraction accepted; lower is better. Second, we measure error correlation and the Q-statistic between base and sidekick, the redundancy measure used by Ensembling Tabular Foundation Models. We compare these numbers against the same quantities for a tiny CNN sidekick and for a second fine-tuned copy of the base. Third, we test whether the retained points concentrate where the base network's held-out errors are, which is the mechanism the method relies on.
As a tabular sanity check in the setting where kernel sparsity originated, we repeat the pruning comparison on Forest/covtype with raw-input kernels. Cross-Training reports 19,526 support vectors at 92.1% accuracy there (Breaking SVM Complexity with Cross-Training). We ask whether that sparse model keeps a useful reject option at all.
Baselines
The naive reference is the base network alone, abstaining by maximum softmax probability, with and without temperature scaling. The selective-classification framing follows Selective Classification for Deep Neural Networks. This baseline costs nothing extra, and any proposal that cannot beat it at matched coverage has failed. On CIFAR-10, Top-Ambiguity Samples Matter reports a single-model AURC of $66.1 \times 10^{-4}$. We will reproduce their setup so that our single-model number can be checked against it.
The strongest accuracy-of-abstention reference is a five-member deep ensemble, together with that paper's cheaper variant that ensembles only the 50% most ambiguous inputs. The same paper reports a deep-ensemble AURC of $51.2 \times 10^{-4}$ on CIFAR-10. This is the strongest concrete number in the file. Closing most of the gap from 66.1 to 51.2 at under 1% extra inference compute would count as a clear win. Ensemble members are trained with the same recipe and data as the base. The held-out split is withheld from them as well, so the comparison is not tilted.
The strongest competitor on the efficiency question is the Asymmetric Duo: a large base plus a tiny architecturally distinct network, combined with validation-tuned temperatures (Asymmetric Duos). It reportedly matches a five-model ensemble on ImageNet at roughly 4× less compute. However, no reported AURC value for Duos was found on Lacuna, so we will run it ourselves. The tiny sidekick network will get the same training data, including the held-out split, and an equal hyperparameter search budget. Beating it means equal or lower AURC, and equal or higher coverage at 98% selective accuracy, at a fraction of its added FLOPs. Matching its AURC at about one-hundredth of its added cost would also count.
A training-side alternative is CCL-SC (Confidence-aware Contrastive Learning for Selective Classification). It reshapes features for selective risk and reports roughly 26.5% error at full coverage on CIFAR-100. We include it to show whether a post-hoc sidekick is competitive with retraining the representation, and whether the two combine.
Three ablations isolate the key ingredient. The first keeps the budget $B$ fixed and compares retained sets chosen by uniform random sampling, by k-center coreset selection, by standard Cross-Training thresholds, and by ISLS-SVM against our disagreement-preserving rule. The second is a dense kernel head with no pruning, which gives the ceiling. The third sets $\lambda = 0$ to separate the sub-machine disagreement signal from the base-error signal. A further ablation fits the same sparse kernel head on features from a different cheap encoder rather than the base's own features. It tells us how much diversity is lost by sharing features.
Why not just…?
Why not just use the Asymmetric Duo with a tiny network sidekick? It is the most direct threat, because it already works and needs no kernel machinery. The argument for our version rests on cost and on a different source of diversity. A few thousand stored points cost far less than 10–20% of a forward pass, and they need no training of a second network. However, if the shared-feature kernel sidekick shows error correlation with the base close to that of a fine-tuned copy, the Duo's architectural distinctness is what matters and our approach loses. The Q-statistic comparison and the head-to-head AURC at matched data settle this. The different-encoder ablation then tells us whether a kernel sidekick on independent cheap features recovers the gap.
Why not just run k-nearest-neighbors on a random subsample of features? This is the obvious skeptic's version of our method. It may capture most of the benefit if the main effect is "any local rule on features helps." That is why random and coreset subsampling at identical $B$ are first-class ablations rather than afterthoughts. If they match disagreement-preserving pruning across budgets, the sparsity contribution is null, and we would report the result as a negative finding about pruning criteria. We expect the gap to appear mainly at small $B$, where random sampling under-represents the thin boundary region.
Why not just calibrate the base model's softmax? Temperature scaling changes confidence values but cannot reorder inputs, so it cannot change which inputs are refused first. CCL-SC argues that output-layer fixes fail when features overlap. "Big VGG-16" in Top-Ambiguity Samples Matter matched ensemble accuracy and still failed at selective classification through overconfidence. We report the calibrated base as the naive reference anyway, because the claim must be shown, not assumed.
Why not just ensemble only the ambiguous half of the inputs? The top-50% rule halves ensemble cost but still requires multiple full networks, and it presupposes members that actually disagree. Agreement-Based Cascading and the tabular diversity-ceiling results show that this presupposition is fragile under shared priors. It remains the accuracy bar we compare against. Our method aims at a different point on the cost curve rather than claiming to dominate it.
Proposed Experiments
The primary benchmarks are CIFAR-10 and CIFAR-100, where reported reference numbers exist, ImageNet-1K for scale, and SVHN, where label noise is known to hurt deep ensembles. The noise is a natural stress test for a rule that preferentially keeps "disagreement" points, some of which will be mislabeled. Covtype provides the kernel-native check. Metrics are AURC, selective risk at fixed coverages, coverage at 98% selective accuracy, accuracy, expected calibration error, base–sidekick error correlation and Q-statistic, and added FLOPs and memory per input.
Four checks are decisive. The first tests whether Gap 1 is real: as the support-vector budget shrinks, Cross-Training and ISLS-SVM should lose substantially more AURC than accuracy. If they do not, the motivation collapses. The second tests whether disagreement-preserving pruning holds AURC near the dense kernel head at budgets of a few percent of the held-out split, while random and coreset pruning do not. The third tests whether the base-plus-sparse-sidekick pair moves CIFAR-10 AURC meaningfully from the single-model value toward the reported ensemble value of $51.2 \times 10^{-4}$, and reaches Duo-level performance on ImageNet at a small fraction of the Duo's added compute. The fourth tests whether the sidekick's error correlation with the base is measurably below that of a fine-tuned copy of the base. Failing the fourth check while passing the third would mean the gain comes from local smoothing rather than diversity, and we would describe it that way.
References
- Breaking SVM Complexity with Cross-Training
- Sparse LS-SVMs with L0 - norm minimization
- Top-Ambiguity Samples Matter: Understanding Why Deep Ensemble Works in Selective Classification
- Asymmetric Duos: Sidekicks Improve Uncertainty
- What is being transferred in transfer learning?
- Quantifying Correlations of Machine Learning Models
- Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- Selective Classification for Deep Neural Networks
- Confidence-aware Contrastive Learning for Selective Classification
- Agreement-Based Cascading for Efficient Inference