Routing the Unfamiliar to "Other": Selective Prediction for Taxonomies with a Catch-All Class

Many real classification schemes contain a leftover label such as "Other," "Junk," "Miscellaneous," or "Dead Coral." That label is defined by what an input is not, rather than by what it is. Selective prediction lets a classifier abstain, meaning it hands a case to a human instead of guessing. The standard theory for it assumes two kinds of hard inputs: ambiguous ones among known classes, which should be deferred, and foreign ones, which should be rejected. A catch-all class breaks this split. In a plankton monitor, a new kind of debris never seen in training is not an anomaly to reject. By the taxonomy's own definition it belongs in "Junk," and the operator wants it labeled that way.

This note argues that in such taxonomies, abstention should be split into two decisions that current methods merge. Inputs that fit no primary class should be labeled "Other." Only inputs that plausibly belong to a primary class, but not confidently to one of them, should be deferred to a human. Making this split well requires estimating how likely an input is to be "Other" from how well it fits the primary classes. It should not come from a softmax output unit trained on whatever residual examples happened to be in the training set, because the makeup of the residual class is exactly what changes at deployment.

Core Claim

The residual class should be modeled as a complement: whatever the stable, compact primary classes do not explain. Once it is modeled this way, the selective prediction rule should treat "predict Other" as a third, cheap action next to "predict a primary class" and "defer."

Concretely, suppose three things hold. First, we have a model of the $K$ primary classes whose fit scores stay meaningful under shift. Second, we have unlabeled deployment data. Third, primary classes drift less than the residual class does. Under these conditions we can re-estimate the probability that an input is "Other" at test time, without any labels for the new residual material.

The expected result is lower cost-weighted error at a fixed deferral budget on shifted deployment data. In particular, we expect far fewer silent errors on container classes, which are the errors Producing Plankton Classifiers that are Robust to Dataset Shift reports as dominating even for its strongest ensemble.

Relation to Prior Work

Two papers motivate the gap by showing where errors land. The plankton study above builds ZooLake2.0 and ten real deployment days. It shows accuracy falling from 92% to 77% under shift, recovered only to 83% by a heavy ensemble, and it finds that errors concentrate on container classes. It never abstains. The coral study Towards highly accurate coral texture images classification using deep convolutional neural networks and data augmentation identifies "Dead Coral" as the main source of confusion, but only qualitatively. Customs classification work Explainable Product Classification for Customs reports depressed confidence on miscellaneous codes even when the prediction is correct. This matters for us: a confidence threshold will then over-defer legitimate "Other" cases.

Three papers supply methods we reuse, and each stops short of our question.

Selective Classification for Deep Neural Networks sets a rejection threshold with a finite-sample risk guarantee from a held-out set. It also shows that the maximum softmax probability is a strong confidence score. We reuse its guarantee machinery, but over three actions instead of two.

Plugin estimators for selective classification with out-of-distribution detection derives the optimal joint rule for deferring ambiguous inputs and rejecting foreign ones: an additive combination of label-error probability and a foreign-to-familiar density ratio. Its loss-based variant trains on unlabeled "wild" data. We keep the additive, density-ratio structure and the use of unlabeled data. The change is in where the density-ratio term sends an input: in our rule it moves probability mass onto the "Other" label, where that paper uses it to trigger rejection. That paper also treats foreign data as external. It never considers a labeled catch-all class inside the training distribution.

Towards Accurate Open-Set Recognition via Background-Class Regularization and its companion Distance-Based Background Class Regularization for Open-Set Recognition replace the softmax layer with distances to per-class anchors. They use background data to squeeze known classes into compact regions, which yields an analytic probability that an input falls inside a class. We use this to build the primary-class fit scores and use training "Other" examples as the background data. Unlike those papers, we then classify the background as a label rather than discarding it into open space.

One paper constrains the design. Hierarchical Selective Classification lets a model retreat to a parent node when it is unsure. It needs a clean tree, and a catch-all class shares no meaningful parent with primary classes other than the root. So "retreat to Other" cannot be expressed as hierarchical climbing. It has to be its own action.

A second constraint comes from Towards Improving Selective Prediction Ability of NLP Systems, which calibrates confidence using how unstable an example was during training. Residual examples are unstable by construction, so a difficulty-based calibrator would push the whole "Other" class toward deferral. We test this failure explicitly rather than assuming it.

Research Program

The system has three parts.

Primary-class model. A distance-based classifier over the $K$ primary classes, trained with background regularization in the style of Cho and Choo, with training "Other" images as the background. It gives two outputs: primary-class posteriors $p(k \mid x, \text{primary})$, and a primary-fit density ratio.

Residual estimator. We train a small discriminator to tell primary-labeled training inputs apart from unlabeled deployment inputs. Deployment data is a mixture: a fraction $\pi_o$ of residual material and $1-\pi_o$ of primary material. If the primary class-conditional distributions are roughly stable, the discriminator's output together with an estimate of $\pi_o$ gives the deployment-time residual posterior

$$\hat{P}(\text{Other}\mid x) = 1 - (1-\hat{\pi}_o)\,\frac{p_P(x)}{p_{\text{dep}}(x)},$$

where $p_P$ is the density of primary-class inputs and $p_{\text{dep}}$ is the density of deployment inputs. We estimate $\hat{\pi}_o$ from the deployment sample in the standard way used when only positive and unlabeled examples are available. The ratio comes from the discriminator; this form follows directly from writing deployment data as that two-part mixture. The residual estimator is combined with the direct residual head of the training-time model only through the learned $\hat{\pi}_o$ weighting. We ablate this combination.

Three-action decision rule. The system chooses

$$a^*(x) = \arg\min_{a \in \{1,\dots,K,\ \text{Other},\ \text{defer}\}} \sum_{y} C(a, y)\, \hat{P}(y \mid x),$$

where $\hat{P}(k \mid x) = (1-\hat{P}(\text{Other}\mid x))\,p(k \mid x, \text{primary})$. The cost matrix $C$ sets a price for each mistake: confusing two primary classes, calling a primary input "Other," calling an "Other" input primary, and deferring at a constant cost. Thresholds are then adjusted on a calibration set with a binomial bound in the style of Geifman and El-Yaniv, giving a guarantee on cost-weighted risk among the non-deferred inputs. The guarantee holds only in-distribution. Under shift we report how often it is violated rather than claim it.

The analysis questions are direct. Does the stable-primaries assumption hold in the real plankton deployment days? We measure drift of primary class-conditional distributions against drift of the residual. How much of the gain comes from the complement estimate, and how much from the three-action rule alone? And do deferrals stop piling up on the "Other" label?

Baselines

Naive reference. A standard softmax classifier with "Other" as an ordinary $(K+1)$-th class, thresholded on maximum softmax probability with the Geifman–El-Yaniv risk bound. Every other method must beat this. On ZooLake2.0 it is trained on the same MobileNet and BEiT backbones as the plankton paper. That paper reports 77% (MobileNet) and 83% (BEsT ensemble) deployment accuracy at full coverage, which anchors the zero-deferral end of every curve. No selective-prediction result on ZooLake2.0 was found on Lacuna.

Strongest existing method. The black-box plug-in estimator of Narasimhan et al., in two configurations: "Other" as an in-distribution class, and "Other" moved to the out-of-distribution side. It is the strongest competitor because it already combines deferral of ambiguous inputs with density-ratio rejection, and it is Bayes-optimal for its own objective. On CIFAR-100 against OpenImages that paper reports an area under the risk-coverage curve of 0.222, against 0.316 for SIRC (a method that multiplies the two scores) and 0.342 for maximum softmax probability. We also run its loss-based variant with the same unlabeled deployment pool our discriminator sees, so neither method gets data the other lacks.

Fairness. All methods share backbones, training data, the unlabeled pool, the calibration split, and a matched hyperparameter search budget.

What counts as beating it. The cost-weighted risk-coverage curve must be lower on at least seven of the ten deployment days, at deferral rates between 10% and 30%. In addition, the fraction of accepted errors that fall on container classes must be clearly smaller.

Ablations. Two ablations isolate the key ingredient. The first keeps the three-action rule but estimates $\hat{P}(\text{Other}\mid x)$ from the training-time softmax head rather than the complement estimator. The second keeps the complement estimator but maps "Other" to deferral, collapsing back to two actions.

Why not just…?

Why not just threshold the softmax with "Other" as a normal class? Narasimhan et al. show that maximum softmax probability can fail badly when ambiguity and shift co-occur, with 0.342 against 0.222 on CIFAR-100/OpenImages. Customs results show residual classes get low confidence even when the prediction is correct, so this baseline should over-defer true "Other" cases and under-defer novel residual material that happens to land confidently in a primary region. Our per-label deferral rates measure both effects directly.

Why not just split "Other" into clusters and train them as sub-classes? This is a real threat. Clustering the residual class into $M$ pseudo-classes removes the single-centroid problem that the Gaussian assumption in background-regularization methods cannot handle. It may well match us on in-distribution tests. It cannot, however, anticipate residual material that did not exist at training time. The held-out-residual experiment below is built to settle this question. If clustered sub-classes match the complement estimator on shifted plankton days, we will drop the complement claim and report only the three-action rule.

Why not just use per-class thresholds? Separate thresholds for "Other" fix unequal deferral across labels, the subgroup-disparity concern the file ties to Plugin estimators for selective classification with out-of-distribution detection. But they are tuned on training-time residual data, which is exactly what drifts. We include per-class Geifman–El-Yaniv thresholds as a baseline. Beating it is required only on shifted data; on in-distribution data we expect a tie.

Why not just ensemble and augment harder? The plankton paper's BEiT ensemble with test-time augmentation still concentrates errors on container classes. The coral paper found augmentation gave under 1% improvement. We run BEsT-style backbones inside both our method and the baselines, so any gain we report comes on top of scaling rather than instead of it.

Proposed Experiments

Controlled residual shift. From CIFAR-100 and ImageNet-1k we choose $K$ primary classes and fold all others into "Other." Half the folded classes are used for training; the other half appear only at test time. Varying the fraction of unseen residual sub-classes from 0 to 100% traces how each method degrades. The complement estimator should degrade least, and the softmax-head ablation most. For iNaturalist 2021 we run the same construction by folding whole subtrees, which also lets us compare against hierarchical climbing on the primary classes.

Real deployment. The main test is ZooLake2.0 with its ten labeled deployment days. The unlabeled pool for each day is that day's own images, with labels hidden. We check the stable-primaries assumption per day by comparing classifier two-sample statistics on primary classes against the residual class. Days where the assumption fails are reported separately, as a predicted failure mode. EILAT and RSMAS provide a small-data check on "Dead Coral," where we expect the cluster baseline to be hardest to beat.

Metrics. Cost-weighted risk against coverage under two cost matrices chosen before any results are seen. We also report the area under the risk-coverage curve, deferral rate per label, the share of accepted errors on the residual class, and how often the risk guarantee is violated under shift.

Decisive outcome. The core claim stands if two things hold. First, on unseen-residual splits and real deployment days, the complement estimator lowers cost-weighted risk against both the plug-in estimator and the clustered-sub-class baseline at matched deferral. Second, the two-action ablation loses most of that gain. It fails if clustering the residual class performs as well under shift.