When the Model Defers on Rare Classes, Do Humans Pick the Common Answer?

AI systems that can say "I'm not sure, please check this" are increasingly built for problems with a long tail. In these problems a few common categories make up most of the data, and hundreds of rare categories each have only a handful of examples. Plant and animal species and rare diseases are typical cases. The usual way to handle the tail is selective prediction: the model answers when it is confident and otherwise hands the case to a person, often with a short list of candidate labels attached. Recent methods tune these candidate lists so that, on paper, rare classes are covered as often as common ones. Nobody has checked what the person receiving the list actually does with it. Work on binary tasks shows that people anchor strongly on whatever the model suggests, and a long-tailed model's suggestions are systematically pulled toward common classes. If that holds here, the human fallback may quietly turn a fair-looking list back into a head-biased decision.

We propose to measure this effect directly and to design deferral around it. On Pl@ntNet-300K, a plant-species benchmark with 1,081 classes, participants resolve the cases the model defers. We vary how the deferral is presented: no help, a flag only, a ranked top-k list, a standard conformal set, or a prevalence-adjusted set, the last of which is built to cover rare classes. We then fit a model of how human accuracy depends on set size, candidate rank, and class rarity. In a second, preregistered study we test whether a deferral policy tuned to that fitted human model beats one tuned to coverage alone.

Core Claim

A candidate set "covering" the true label is not the same as the human choosing it, and the gap between the two depends on class frequency. When the true species is rare and appears in a list alongside common look-alikes, people will disproportionately choose the common one, especially when it is ranked first. We call this the set-to-selection gap. We predict it is small for head classes and large for tail classes.

The accounting makes the claim concrete. For a class $y$, team accuracy is

$$A_y = c_y\, a^{\text{AI}}_y + (1-c_y)\, h_y(D),$$

where $c_y$ is the fraction of class-$y$ items the model answers itself, $a^{\text{AI}}_y$ is its accuracy on those, and $h_y(D)$ is the human's accuracy on deferred class-$y$ items under display condition $D$. The quantity we care about is the unweighted mean of $A_y$ over classes. Current long-tail deferral methods implicitly set $h_y(D)$ equal to "true label is in the set" or to a constant. Our claim is that $h_y(D)$ falls off with rarity and rank in a measurable way. Once it is estimated, the best set-construction rule changes, so that coverage-optimal sets are not team-optimal sets.

Prior Work and What It Leaves Open

Three algorithmic papers motivate the question and supply the methods we reuse.

Conformal Prediction for Long-Tailed Classification shows two failure modes on Pl@ntNet-300K and iNaturalist-2018. Standard conformal sets meet their overall coverage target by giving up on rare classes. Per-class conformal sets cover the tail but grow very large. Its fix, the Prevalence-Adjusted Softmax score (PAS), divides each class probability by that class's frequency, so that the average coverage across classes is high and sets stay small. We use PAS as our strongest algorithmic condition. We differ in measuring whether its coverage survives contact with a person, which that paper does not test.

Act or Defer splits clinical decisions into an error-controlled "act" arm and a deferral arm that returns structured sets. It reports that naive conformal selection on pancreatic cancer gives only about 10% selected coverage for the positive class. It assumes that deferred sets are adjudicated correctly and includes no clinician study.

Trustworthy Long-Tailed Classification shows that softmax confidence is overconfident on tail classes, while evidence-based uncertainty tracks error better. This tells us the model's candidate list will be head-skewed exactly where deferral happens.

Two human studies motivate the anchoring hypothesis and also constrain the design. Role of Human-AI Interaction in Selective Prediction ran a 2×2 study with 198 participants on binary Snapshot Serengeti camera-trap images. Human accuracy on deferred items was 58.4% with no message, 61.9% when told only that the AI deferred, and 57.8% when shown the AI's tentative label. Accuracy fell to 41.9% when that label was wrong. Know What Not To Know found that abstention did not hurt perceived competence. User accuracy was 69% with an abstaining AI against 42% with a wrong AI.

Both studies are binary and balanced, and that matters. In a two-class task, "just tell them it deferred" is a complete interface. With 1,081 species, a non-expert told only "the model is unsure" faces an unaided search. Even expert botanists in Pl@ntNet-300K image dataset erred 20.5% of the time while naming about 4.1 candidates per image. So the many-class tail forces a question the binary work could not ask: which list to show, not whether to show one.

Trust, distrust, and appropriate reliance in (X)AI and Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans set our evaluation standard. The first shows that self-reported trust does not predict how much people actually rely on a system. The second shows that fewer than 1% of explainability papers test with humans at all. We therefore measure behavior, not Likert trust, and we treat any simulated human as a hypothesis to be validated.

Research Program

Phase one is a measurement study on Pl@ntNet-300K, with an iNaturalist-2018 subset as a replication.

We train two classifiers. The first is a plain cross-entropy model. The second is trained with the Label-Distribution-Aware Margin loss, a long-tail method that gives rare classes larger decision margins. On each we set a confidence threshold that defers a fixed fraction of test items. We then build deferral displays for the same deferred items.

Every list shows a reference photo for each candidate species, mimicking field-guide apps. Participants pick one candidate or choose "none of these," which opens a searchable genus catalog. Trials are stratified so each participant sees deferred items from head, middle, and tail frequency bins. The model's own answered items are scored automatically.

We log five quantities:

  • the participant's choice;
  • the rank of the chosen candidate;
  • whether the true label was in the list;
  • time to decision;
  • for wrong answers, the training frequency of the chosen class.

The last of these gives a "head-pull rate": among tail-item errors, the share that land on a more frequent class.

From these data we fit a human-response model. It is a logistic model of picking correctly given that the truth is in the set, as a function of set size, rank of the true label, the frequency bin of the true label, and the frequency gap to the top-ranked candidate.

Phase two uses the fitted model to choose, per frequency bin, the threshold and list ordering that maximize predicted team class-averaged accuracy under a fixed human-workload budget. We preregister the predicted gain over PAS and test it on fresh participants and fresh items. If the fitted model's predictions miss by more than its own uncertainty, that is itself a finding: human behavior on the tail would then be harder to model than our claim assumes.

We recruit crowd participants for both phases. We also recruit a smaller cohort of experienced naturalists for a within-subject version of the key contrasts. This addresses the convenience-sample criticism in the 1% review.

Baselines

Naive reference points. The first is no AI help: the participant identifies the deferred image from the catalog alone. The second is the flag-only condition that won in Bondi et al. These anchor everything else.

The flag-only result cited from the binary setting is 61.9% human accuracy on deferred Snapshot Serengeti items, against 58.4% with no message. To tie our numbers to theirs, we rerun a small binary Serengeti arm with identical conditions. If we fail to reproduce their direction, our interface rather than the tail is the likely cause, and we fix that before interpreting Pl@ntNet results.

On Pl@ntNet, beating flag-only means higher class-averaged team accuracy at equal or lower median decision time. Lists trivially help with search, so time is the fairness control.

Strongest existing method. This is a PAS conformal set from Conformal Prediction for Long-Tailed Classification. It is calibrated on the same calibration split and the same backbone, targets 90% class-averaged coverage, and is shown in PAS-score order.

That paper reports PAS as Pareto-optimal for set size against class-averaged coverage on Pl@ntNet-300K. No reported numerical set size on Pl@ntNet was found on Lacuna, and no reported result was found on Lacuna for human accuracy with any conformal set on a long-tailed benchmark. We will report PAS's algorithmic coverage alongside its human-realized accuracy.

Our phase-two policy beats PAS if its class-averaged team accuracy on the tail bin is higher. The sample-size calculation is fixed in advance from phase-one variance. Head-bin accuracy must not drop by more than a preregistered margin, and expected human workload must be matched.

Other comparisons. We include a softmax top-k list, with k matched to PAS's mean set size, and a standard marginal conformal set. These separate the effect of the scoring rule from the effect of list length.

Ablation of the key ingredient. The key ablation shows the PAS set's candidates in alphabetical order, with no rank. If the set-to-selection gap is driven by rank anchoring toward head classes, randomizing order should shrink the tail gap without changing list contents. If it does not shrink, the gap comes from familiarity with common species rather than from the model's ordering. In that case the fix lies in what is shown, not how it is ordered.

Backbone ablation. Comparing the plain model with the long-tail-trained model tests whether better training alone removes the head pull.

Why not just…?

Why not just show "the AI deferred" and nothing else, since that won in the binary study? In a 1,081-class task, a flag alone leaves the person with an unaided search that even botanists find hard: 20.5% error while hedging over about 4 candidates, per Pl@ntNet-300K image dataset. We include flag-only as a full condition, so if it wins on class-averaged accuracy per minute, the proposal's answer is simply "don't show lists." That would be a real and publishable result about how binary findings fail to transfer.

Why not just use PAS sets, which already fix the tail? PAS fixes the tail in coverage, which is the probability that the truth is in the list. Our hypothesis is that humans convert coverage into correct choices less efficiently for rare truths. This is a direct, falsifiable threat to the proposal. If PAS's set-to-selection gap turns out to be flat across frequency bins, phase two has nothing to improve, and we will report PAS as human-validated instead.

Why not just train a better long-tailed model and threshold its softmax? Better training narrows the head bias in the candidates. It does not remove softmax overconfidence on the tail documented in Trustworthy Long-Tailed Classification, and it does not touch human familiarity bias. The LDAM backbone condition measures how much of the head-pull rate disappears with better training. If most of it does, the recommendation becomes "fix the model," and the display question becomes secondary.

Why not just simulate the human with a fixed accuracy? The binary studies already show that human accuracy swings from roughly 42% to 76% depending on what is displayed. A constant cannot represent that. We test the alternative explicitly. We choose a deferral policy under a constant-accuracy simulator, choose one under our fitted frequency-aware model, and compare both against real phase-two outcomes.

Decisive Checks

The project turns on three results.

  1. The set-to-selection gap, defined as coverage minus human accuracy given the truth is in the set, must grow from head to tail in phase one. If it is flat, the core claim is false.
  2. On tail-item errors, the head-pull rate under ranked lists must exceed the rate under alphabetized lists. This decides whether ordering is the mechanism.
  3. In phase two, the policy chosen with the fitted human model must beat the PAS-for-coverage policy on tail-bin team accuracy at matched workload. It must also beat the policy chosen under a constant-accuracy simulator.

The naturalist cohort checks whether experts, who know rare species exist, show a smaller gap. That result would tell practitioners whether the fix belongs in the interface or in who receives the deferral.