Most selective classifiers are built for plain accuracy. A selective classifier is a model that may say "I don't know" and pass an input to a human. It is normally judged by how its error rate on the inputs it keeps falls as it keeps fewer of them. That setup assumes each kept input adds its own independent error. Many real deployments are instead scored with corpus-level metrics such as micro-F1 or macro-F1 over a multi-label tagging task. Medical indexing, clinical note coding and document tagging are typical cases. Under these metrics the value of keeping one input depends on every other kept input through a shared confusion matrix. This note claims that the right rule for which inputs to abstain on, under such metrics, is not a confidence score. It is an expected marginal utility whose weights depend on the metric value the system reaches on the kept set. That rule can be derived in closed form for the linear-fractional family of metrics (F1, Jaccard, and ratios of weighted confusion counts), computed from ordinary per-label probabilities, and calibrated with a finite-sample guarantee.
The practical point is simple. Under micro-F1, true negatives never enter the score. A confidently all-negative document therefore contributes nothing to the metric, even though a confidence-based rule would keep it first. A document with one uncertain positive, by contrast, can pull the kept-set F1 down. The two rules rank inputs differently, so they produce different risk-coverage curves. They also expose a gaming risk that no selective-prediction paper in our file discusses. This proposal builds the metric-aware selector, proves what it optimizes, and tests whether the difference matters on standard multi-label benchmarks.
Core Claim
Two results supply the method. Consistent Multilabel Classification proves that for linear-fractional metrics like micro-F1, the Bayes-optimal forced-prediction rule thresholds every marginal label probability $\eta_m(x)=P(Y_m=1\mid x)$ at one shared value. F1-Optimal Thresholding in the Multi-Label Setting shows that for calibrated probabilities this value is $F^*/2$, half the best achievable F1. Both results assume every input must be labelled.
We add a selection function $g(x)\in\{0,1\}$ with a coverage constraint $\mathbb{E}[g(x)]=c$. Micro-F1 on the kept set is a ratio of two sums of per-example terms, so the standard trick for maximizing a ratio applies. At the optimum value $F^*_c$, maximizing $F$ is equivalent to maximizing the linear quantity $2\,TP - F^*_c(2\,TP+FP+FN)$. A Lagrange multiplier $\lambda$ on coverage then yields the rule below.
Under this rule, each example is kept if and only if its expected contribution to that linear quantity clears $\lambda$:
$$u_c(x)=\sum_{m:\,\hat y_m=1}\big[(2-2F^*_c)\,\eta_m(x)-F^*_c\,(1-\eta_m(x))\big]\;-\;\sum_{m:\,\hat y_m=0}F^*_c\,\eta_m(x)\;\ge\;\lambda .$$
Here $\hat y_m$ is the predicted label. The prediction rule on kept examples stays a shared threshold, $\hat y_m=1$ iff $\eta_m(x)>F^*_c/2$. The first sum is the expected true-positive gain minus the false-positive penalty over labels predicted positive. The second sum is the expected false-negative penalty over labels predicted negative.
This answers the open question in the file's Gap 2: shared-threshold optimality survives selection. The threshold moves upward, however, because $F^*_c$ rises as coverage falls. The selector also ranks inputs by $u_c$, not by confidence. The same derivation holds for any linear-fractional metric with different weights. For plain accuracy it reduces to thresholding the maximum class probability, which is exactly the Softmax Response rule of Selective Classification for Deep Neural Networks. The accuracy case is therefore a sanity check, not a separate method.
Relation to Prior Work
Two bodies of prior work motivate this proposal from opposite sides. The selective-prediction line keeps risk decomposable. Its members are the risk-guarantee procedure of Selective Classification for Deep Neural Networks, the three-head network of SelectiveNet, the training recipe of Towards Better Selective Classification, and the cross-fitted wrapper of A Model-Agnostic Heuristics for Selective Classification. The last of these names non-additive losses as an open extension. The metric-agnostic multi-label line handles arbitrary metrics but always labels everything. Its members are CLEMS, which embeds label sets so distances match the metric; LLSL, which fits local surrogates of a black-box metric; and RethinkNet, which refines predictions with a cost-weighted recurrent loss.
The plug-in consistency results of Koyejo et al. and Narasimhan et al. supply the method. We reuse their "estimate probabilities, then solve a low-dimensional threshold problem" structure. We add one extra dimension, the acceptance cutoff.
The closest bridging works differ in kind. SelMix uses a first-order gain estimate of a non-decomposable metric to choose which class pairs to mix during training. It never abstains at test time. Our $u_c$ is a gain estimate used for the opposite purpose, deciding what to withhold. Hierarchical Selective Classification replaces rejection with coarser predictions in a class tree and does not address corpus-level metrics.
Research Program
The work has three parts.
The first part is the theory. We derive the Bayes-optimal selective rule for general linear-fractional metrics under instance-level abstention. We then extend it to cell-level abstention, where the system withholds individual (instance, label) decisions. The extension targets macro-F1, where each label has its own $F^*_{c,m}$ and weights. Here we test the file's Gap 3 directly. Lipton et al. showed that an uninformative rare label is best predicted positive everywhere under forced macro-F1. On MeSH this meant predicting rare terms for more than a third of documents. The derived rule should instead withhold those cells, because their $u$ is negative once $F^*_{c,m}$ can exceed the tiny all-positive F1.
The second part is the estimator. The estimator takes probabilities from any base model: binary relevance logistic models, gradient-boosted trees, or a deep multi-label network. It solves for $(F^*_c,\lambda)$ by a nested one-dimensional search on a held-out split, following the two-split plug-in recipe that Narasimhan et al. prove consistent.
The third part is the guarantee. SGR inverts a binomial bound because per-example errors are independent Bernoulli events. That fails for F1, as the file's Gap 6 notes. However, kept-set micro-F1 is a ratio of two means of independent, bounded per-example quantities, $2TP_i$ and $2TP_i+FP_i+FN_i$. We therefore bound the numerator from below and the denominator from above with a union bound. This yields a finite-sample lower confidence bound on the kept-set metric. We then pick the largest coverage whose bound clears a user target $F_{\text{target}}$ with probability $1-\delta$. This bound is looser than a binomial inversion, and we will report how much coverage it costs relative to an oracle.
Finally, the program examines the metric's gameability. The derived selector will happily reject confident negatives, because they add nothing to micro-F1. We will report this openly. We will also study a second objective in which rejected examples go to an expert who labels them correctly at a fixed cost per example. Under that system-level objective confident negatives are no longer free to discard, which gives a deployment-honest variant of the same derivation.
Baselines
The naive reference point is forced prediction with the Koyejo shared threshold tuned at full coverage. It has no abstention. Every selective method must beat its micro-F1 at 100% coverage while keeping fewer examples. A second naive point pairs the same fixed threshold with random rejection, which shows what coverage alone buys.
The strongest existing competitor is post-hoc confidence selection with the prediction threshold re-tuned on the kept set. We rank examples by the smallest margin $\min_m|\eta_m(x)-t|$ and alternatively by mean label entropy. We keep the top fraction and then re-tune the shared threshold on the kept validation examples. This is the multi-label analogue of Softmax Response. Selective Classification for Deep Neural Networks found Softmax Response competitive with Monte Carlo dropout, and Towards Better Selective Classification found it beat specialized selection heads when paired with good training. It is therefore the baseline most likely to match us.
A third baseline, also strong, scores each example by its expected instance-level F1 and keeps the highest. Instance-averaged F1 decomposes over examples, so this is the principled selector for that metric. It tests whether corpus-level structure matters at all.
We also run the SCROSS wrapper with the margin score on the tabular-feature datasets, and a SelectiveNet-style multi-label network with a sigmoid selection head trained per target coverage.
Two ablations isolate the key ingredient. The first uses $u$ with $F^*$ frozen at its full-coverage value, which tests whether re-solving $F^*_c$ matters. The second uses our selector with the prediction threshold fixed at 0.5, which tests whether the shift to $F^*_c/2$ matters.
All methods share the same base probability model, trained once per dataset, and the same train, validation and test splits. They also get the same validation data for every threshold and cutoff search, so the comparison measures selection rules, not representation quality. SelectiveNet is the exception because it retrains. It gets the same number of epochs and hyperparameter trials as the base model. Beating the confidence baseline means a higher area under the micro-F1 versus coverage curve, and higher kept-set micro-F1 at 90%, 80% and 70% coverage. The gap must exceed the spread across five random splits, and it must hold on a majority of datasets.
The file gives no reported result on Lacuna for selective micro-F1 or selective macro-F1 on any multi-label benchmark. For forced-prediction multi-label numbers on these datasets it also gives no reported result on Lacuna for LLSL, CLEMS or RethinkNet, since only average ranks are summarized. The only anchor figure concerns the accuracy special case, where SGR reports a 1% error rate at 78% coverage on CIFAR-10. We will reproduce this to confirm that our code reduces to SGR when the metric is accuracy.
Why not just…?
Why not just threshold confidence after training a good metric-aware model? This is the real threat. On datasets where confident examples happen to be the ones with confident positives, the two rankings will nearly coincide. The derivation predicts where they diverge: data with many near-empty label sets, such as enron, medical and MeSH-style indexing, where confident negatives are common. We will measure rank correlation between $u_c$ and the margin score per dataset and relate it to the gain. If the confidence baseline matches us everywhere, the conclusion is that confidence suffices, and the theory still explains why.
Why not just give each label its own reject option, as in binary relevance? Independent per-label abstention ignores the shared denominator of micro-F1. Multi-Label Learning with Stronger Consistency Guarantees shows binary relevance surrogate bounds loosen as $\sqrt{l}$ in the number of labels. Our cell-level macro-F1 experiment runs per-label SGR on each label as a direct comparison, reporting both the metric and the rare-label false-positive rate studied by Lipton et al..
Why not just train better, as with self-adaptive training plus entropy minimization? Better training improves the probabilities that our selector consumes. It does not change which rule should rank them. We apply the same training recipe to the base model for every selector, so any remaining gap belongs to the selection rule.
Why not just search thresholds and cutoffs to maximize validation F1 directly? That is what our estimator does in two dimensions. The danger is overfitting small validation sets, which the file calls the winner's curse. We compare unconstrained empirical search with our bound-calibrated choice, recording how often test F1 falls below target.
Decisive Checks
The benchmarks are the multi-label suites used by CLEMS, LLSL and Koyejo et al.: emotions, scene, yeast, birds, CAL500, enron, medical and tmc2007. We add MSCOCO for a deep base model and Medline/MeSH for the macro-F1 rare-label study. CIFAR-10 serves as the accuracy sanity check.
Four results would settle the proposal. First, the metric-aware selector must beat re-tuned confidence selection by a margin beyond split noise on the sparse-label datasets. The gap should shrink where $u_c$ and confidence are highly rank-correlated. Second, the learned prediction threshold should track $F^*_c/2$ as coverage falls, and freezing it should hurt measurably. Third, on MeSH, cell-level abstention should cut the fraction of documents tagged with uninformative rare terms far below the one-third Lipton et al. report, while matching or exceeding forced macro-F1 on kept cells. Fourth, the ratio bound should keep test F1 above target in at least a $1-\delta$ fraction of resampled splits, while naive search violates it more often. For each check we report the coverage the guarantee gives up.
We will also report the system-level objective, with expert handling of rejected examples, alongside kept-set F1. A reader can then see whether the gains come from genuinely hard examples or from discarding easy negatives.
References
- Consistent Multilabel Classification
- F1-Optimal Thresholding in the Multi-Label Setting
- Selective Classification for Deep Neural Networks
- SelectiveNet: A Deep Neural Network with an Integrated Reject Option
- Towards Better Selective Classification
- A Model-Agnostic Heuristics for Selective Classification
- Cost-sensitive label embedding for multi-label classification
- A Deep Model With Local Surrogate Loss for General Cost-Sensitive Multi-Label Learning
- Deep Learning with a Rethinking Structure for Multi-label Classification
- On the Statistical Consistency of Plug-in Classifiers for Non-decomposable Performance Measures
- Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectives
- Hierarchical Selective Classification
- Multi-Label Learning with Stronger Consistency Guarantees