The Price of Abstaining on Rare Classes: An Axiomatic Theory of Selective Prediction Metrics

A selective classifier is a model that may say "I don't know." It either returns a label or defers the input to a human or a fallback system. We usually score such models with a risk-coverage curve, which plots the error rate on the inputs the model accepts against the fraction of inputs it accepts (the coverage), and we summarize that curve by its area (AURC). This note argues that on imbalanced data the curve can be gamed in a specific way that no current selective metric both detects and corrects. A model can look better by quietly refusing whole rare classes. Refusing a class that makes up 2% of the data costs only 2% coverage, but under any chance-corrected, class-balanced notion of error it removes a full class's share of the error.

We propose to show that this is not a defect of any one metric but a genuine trade-off, analogous to the impossibility theorems already proven for classification and clustering measures. We conjecture that no selective metric can satisfy all three of the following. The first is chance correction under class imbalance (a constant baseline). The second is coverage faithfulness: the coverage axis means the actual fraction of inputs automated, which is the quantity operational budgets are written in. The third is abstention neutrality: the score cannot be improved by concentrating rejections on small classes. The deliverables are three. We would prove this trilemma, place every metric in use (AURC, Excess-AURC, the selective accuracy constraint, selective AUC, and their class-balanced variants) at a specific corner of it, and give practitioners a principled two-number report with a matching training objective.

Core Claim

Classification measures have an axiomatic theory. Good Classification Measures and How to Find Them proves that no full-coverage measure is simultaneously monotone, a distance, and chance-corrected. It recommends Symmetric Balanced Accuracy, which gives up only the distance property. That paper explicitly excludes models that abstain. Selective prediction, meanwhile, has a strong statistical theory of its main metric. A Novel Characterization of the Population AURC shows that population AURC equals a reweighted 0/1 risk:

$$\text{AURC} = \mathbb{E}_{(x,y)}\left[-\ln(1-G(x))\,\ell(f(x),y)\right],$$

Here $G(x)$ is the percentile rank of the input's confidence and $\ell$ is the 0/1 error. The weight depends only on confidence, never on class, so AURC inherits every imbalance pathology that Gösgens et al. proved for plain accuracy. Zhou et al. list imbalance as open.

What changes once abstention enters is the exchange rate between coverage and error. Let $\pi_k$ be the population share of class $k$ and $K$ the number of classes. Suppose error is balanced across classes, so each class contributes weight $1/K$, and coverage is operational. Then rejecting all of class $k$ costs $\pi_k$ coverage and removes a $1/K$ share of the error. For rare classes $\pi_k \ll 1/K$, so a monotone metric must reward the move. You can remove the incentive in two ways. You can measure coverage in balanced units too, which amounts to reweighting the data to a uniform class prior, but then the coverage axis no longer means "fraction of inputs automated." Alternatively you can keep plain error, but then you lose chance correction. This is the conjectured trilemma. The new content is the formal statement, a proof, and a map of existing metrics onto its corners. The heuristic argument above is not itself new.

Relation to Prior Work

Three papers motivate the gap. AUC-based Selective Classification observes empirically that error-optimizing selective models such as SelectiveNet reach their coverage targets by rejecting minority positives, for example on the CSDS2 fraud data with 1.8% positives. Its fix, AUCROSS, rejects an interval of scores around the decision threshold to raise selective ranking quality. That is an algorithm for one binary metric, not an account of why metrics disagree. How to measure deep uncertainty estimation performance shows across 484 ImageNet models that AURC, AUROC, the selective accuracy constraint and calibration error rank models inconsistently. It offers no theory, and it evaluates only on balanced data, which is exactly where our trilemma predicts the corners coincide. Fair Selective Classification Via Sufficiency and Towards Better Selective Classification note that lowering coverage can discard particular subgroups. Our abstention-neutrality axiom formalizes the class-level version of that concern.

Other work supplies methods. We reuse the axiomatic proof technique of Gösgens et al., which constructs small confusion configurations where two properties force contradictory orderings, together with its cluster-index predecessor Systematic Analysis of Cluster Similarity Indices. We also reuse Zhou et al.'s reweighted-risk view and its fast sorting-based estimators, extending them to class-dependent weights. Measuring Informativeness Gap of (Mis)Calibrated Predictors plays a constraining role. Its continuity trilemma for decision informativeness warns that chance-corrected, complete orderings may be discontinuous, so we test whether our chosen corner metric is continuous in the confidence scores rather than assuming it. Hierarchical Selective Classification lets a model retreat up a taxonomy instead of abstaining entirely. We treat it as a later extension and keep the core claim flat.

Research Program

The theoretical part defines a selective measure as a function of the joint table of true class, predicted class, and accept/reject, evaluated either at a threshold or integrated over thresholds. It then states six axioms: the three above, plus coverage monotonicity (lower error at equal coverage, or higher coverage at equal error, strictly helps), bounds, and class symmetry. We will attempt to prove the trilemma by constructing a two-class instance in which chance correction and coverage faithfulness pin down the exchange rate and a class-drop selector then strictly improves the score.

If the trilemma turns out false as stated, we will report the tightest weakening that holds. One possibility is that it holds only for threshold-integrated measures. A counterexample would itself be informative, because it would identify a metric satisfying all three axioms.

We then classify candidate metrics against the axioms. Plain AURC and Excess-AURC are coverage-faithful and neutral but not chance-corrected. Importance-weighted AURC, which reweights to a uniform prior, is chance-corrected and neutral but not coverage-faithful. Per-class averaged AURC, selective Symmetric Balanced Accuracy at a fixed coverage, and selective AUC each get a placement. We expect selective accuracy with balanced error but operational coverage to be the gameable corner.

Finally we extend Zhou et al.'s estimator to class-dependent weights. This gives a trainable objective for whichever corner a practitioner chooses, and we prove the same $O(\sqrt{\ln n / n})$ consistency rate carries over when class priors are estimated. The rate should survive because prior estimation adds only an $O(1/\sqrt{n})$ term.

Baselines

The do-nothing reference is the standard pipeline. A cross-entropy model's maximum softmax probability is thresholded (Softmax Response, from Selective Classification for Deep Neural Networks) and scored by AURC. On CIFAR-10 that paper reports 1% error at 78% coverage with its guaranteed-risk procedure. This is the reference point every metric is computed on.

The strongest existing model-side competitor is direct AURC fine-tuning with Zhou et al.'s estimator. It reports AURC of 6.007×10⁻² on CIFAR-100 with ResNet-50, versus 6.225×10⁻² for cross-entropy training. We rerun it on the same backbones, data splits and tuning budget as our class-weighted variant. Beating it means one of two outcomes. Either we get lower minority-class rejection and lower balanced selective error at matched operational coverage on imbalanced data, without losing more than its own seed-to-seed variation in plain AURC on balanced CIFAR-100. Or, if that trade-off is unavoidable, we show it follows the curve the theory predicts.

The strongest imbalance-aware competitor is AUCROSS and its deep version from Pugnana and Ruggieri, run with their cross-fitting on LendingClub and CSDS2. The file reports no exact AUC values for AUCROSS on CSDS2, so we will reproduce its numbers ourselves before comparing. We also include SAT with entropy minimization plus Softmax Response from Towards Better Selective Classification, which the file reports beats SelectiveNet on several benchmarks (no specific AURC numbers are recorded in the file), and SelectiveNet itself as the method known to drop minority classes.

The key ablation keeps the risk axis class-balanced and switches only the coverage axis between operational and balanced units. This isolates the exchange-rate mechanism the trilemma is about.

Why not just…?

Why not just reweight the data to a uniform class prior and compute ordinary AURC? This is the real threat, and we expect it to satisfy chance correction and neutrality. The theory predicts its cost precisely: its coverage axis is no longer the fraction of inputs automated. A team with an 80% automation budget reading off 80% balanced coverage may actually be automating a quite different fraction. We will measure that gap directly on CSDS2 and long-tailed CIFAR. If the gap is negligible at practical operating points, we will say so. The practical recommendation then becomes this reweighting, and the theorem's contribution shrinks to explaining why it is the right corner.

Why not just use selective AUC, as AUCROSS does? Selective AUC is binary and measures ranking quality on the accepted set, not errors at the deployed threshold. The file notes that Pugnana and Ruggieri leave multiclass open. We will place it in the axiom map and test it with the class-drop selector described below. Our prediction is that it is chance-corrected but not a measure of operational risk, so it answers a different question.

Why not just apply temperature scaling to Softmax Response? Temperature scaling changes the model, not the metric. It can reorder confidences across inputs, which is consistent with Galil et al.'s finding that it improves AUROC. It does not change what any metric rewards. We will still report whether it shifts minority rejection rates, since InfoGap's "utility harm" result suggests sharpness loss may fall unevenly on rare classes.

Why not just report per-class risk-coverage curves? Per-class curves are the ground truth, but they do not rank models. Averaging them requires a choice of coverage units, and that choice is exactly the trilemma. We will use the per-class curves as the diagnostic against which every scalar is checked.

Proposed Experiments

The decisive check is an adversarial one. For each trained model we construct a class-drop selector that keeps the model's predictions but additionally rejects the smallest class, then the two smallest, and so on. We record which metrics score it higher than the original.

The theorem predicts a specific pattern. Every coverage-faithful, chance-corrected metric should reward the class-drop selector. Plain AURC should not reward it, but should fail a random-predictor test. In that test, a random predictor with a random selector should yield the same score as class skew varies, and plain AURC's score should drift with skew. Importance-weighted AURC should resist the class-drop selector while shifting the operational coverage it reports. If any metric escapes all three failures, the conjecture is wrong.

We run this on CSDS2 and LendingClub from Pugnana and Ruggieri, SST-5 from Gösgens et al., and CIFAR-10/100 subsampled to controlled long-tail ratios. We also run on balanced ImageNet as a control, where all corners should agree up to noise. That agreement would explain why Galil et al.'s balanced benchmark never surfaced the issue.

We then measure how much metric choice reorders models, computing Kendall rank correlations between metrics over a fixed pool of Softmax Response, temperature-scaled, SAT+EM, SelectiveNet, AUCROSS, Zhou-fine-tuned and class-weighted-fine-tuned models. We expect correlations near one on ImageNet and markedly lower as imbalance grows.

Finally we train with the class-weighted estimator. We check that it reduces minority rejection at matched operational coverage relative to Zhou et al. and AUCROSS. We also check that it is not discontinuous in the scores, by perturbing the logits slightly and tracking score changes, which is the failure InfoGap's trilemma warns of.