Why Allowing Models to Abstain Magnifies Bias

64 concepts35 papers2 proposals

In high-stakes machine learning applications, a model's ability to say "I don't know" is often framed as a vital safety feature. This approach, known as selective classification or prediction with a reject option, allows a classifier to abstain from making a prediction when its internal confidence falls below a certain threshold. The intuitive appeal is straightforward: by weeding out low-confidence, error-prone decisions and routing them to human experts, we should be able to boost the accuracy and reliability of the remaining automated decisions.

However, a growing body of research reveals a critical failure mode in this logic. Far from being a neutral tool for risk management, selective classification frequently introduces, shifts, and magnifies demographic disparities. As Jones et al. demonstrated, pruning low-confidence predictions often widens the performance gap between majority and minority groups, sometimes even reducing the absolute accuracy for the most vulnerable populations.

To understand why this happens, we must look beneath aggregate performance metrics and examine how model uncertainty interacts with systemic bias.

The Core Failure Mode: Miscalibrated Confidence Across Groups

The fundamental reason selective classification fails to deliver equitable outcomes is that a model’s internal confidence is rarely distributed uniformly across different social groups.

Machine learning models are typically optimized on data dominated by majority group patterns. Consequently, the model learns the features of the majority population far better than those of marginalized subgroups. When a global confidence threshold is applied across the entire dataset, it acts as an unintended demographic filter:

  1. Disproportionate Rejection (Service Availability Bias): Because the model is systematically less certain about underrepresented populations, it abstains from predicting on them far more frequently. Pugnana et al. observe that this creates an inequitable distribution of service availability, effectively locking minority groups out of automated decision-making benefits.
  2. The Minority Class Rejection Bias: In a large-scale benchmark of deep neural networks, Pugnana et al. found a pervasive hidden bias where selective classification algorithms disproportionately reject instances belonging to minority classes, leading to systemic exclusion if the deferred cases are not carefully managed.
  3. Vulnerability to Distribution Shifts: When real-world data shifts away from the training distribution, these shifts rarely happen uniformly. Goel et al. note that if distribution shifts occur primarily within specific demographic subgroups, a standard selective prediction mechanism will respond by disproportionately refusing to serve those exact populations.

The Paradox of Risk Inflation

One might assume that even if a minority group faces a higher rejection rate, the remaining accepted predictions for that group would at least become highly accurate. Remarkably, this assumption is mathematically and empirically false.

In a study on selective regression, Shah proved that as a system becomes more selective (i.e., lowering coverage to boost average accuracy), the error rate for minority subgroups can paradoxically increase.

This occurs because the uncertainty metric itself is optimized for majority data patterns. A minority sample that looks "certain" to a majority-biased model might actually be highly prone to error. Furthermore, Johansson et al. showed that standard calibration techniques break down in high-rejection scenarios; when a model is allowed to reject difficult cases, the reported confidence of the remaining predictions often completely decouples from observed precision.

Confident but Biased: The Blind Spot of Uncertainty

A secondary, deeper issue is that high model confidence does not guarantee an unbiased or ethical outcome. Classifiers easily learn historical prejudices or biased proxy features from training data and output them with absolute mathematical certainty.

As Ni et al. point out, high-certainty predictions frequently rely on these biased correlations, whereas samples where the model is less certain often exhibit more equitable performance. Because standard selective classification relies entirely on uncertainty metrics, it is completely blind to instances where a model is confidently wrong due to demographic prejudice (Pugnana et al.). The system happily passes through highly confident, biased decisions while rejecting the nuanced cases where equitable trade-offs could actually be made.

Designing Fairness-Aware Abstention

Recognizing that group-agnostic abstention magnifies bias, researchers have begun developing frameworks that explicitly integrate fairness constraints into the rejection logic.

Post-Processing for Demographic Parity

Instead of letting a model naturally abstain based on raw confidence, we can use abstention as a deliberate tool to enforce fairness. Schreuder et al. introduced a post-processing framework that strategically applies a rejection option to an existing classifier to eliminate disparate impact. They demonstrated that rejecting as little as 1% to 5% of carefully selected samples can satisfy demographic parity while preserving or even improving accuracy on the remaining predictions, effectively bypassing the traditional risk-fairness trade-off.

Bypassing Mathematical Impossibility

In standard binary classification, it is a well-known mathematical reality that multiple fairness metrics (such as predictive parity and equalized odds) are mutually incompatible unless the base rates across groups are identical. However, introducing a neutral "abstain" state fundamentally changes this geometry. Canetti et al. proved that by conditioning fairness metrics only on the subset of cases where a definitive decision is made, a system can simultaneously satisfy multiple statistical parity constraints that are otherwise mathematically incompatible.

Explicit Fairness Audits in the Loop

Rather than relying on a single confidence score, modern frameworks like the Interpretable and Fair Abstaining Classifier (IFAC) combine uncertainty metrics with explicit, rule-based fairness checks (Pugnana et al.). If a prediction is mathematically certain but triggers a local or global fairness violation, the system overrides the confidence score and routes the case to a human expert.

Adaptive and Long-Term Perspectives

Recent work also emphasizes that fairness interventions should be applied selectively where disparities actually exist, rather than exhaustively across all demographic slices, to maintain predictive efficiency (Zhou et al.). Furthermore, when evaluating the long-term impact of these systems, Guldogan et al. argue that fairness audits must focus heavily on the rejected population to ensure that individuals deferred to human or secondary processes retain an equal opportunity to qualify for automated acceptance in the future.

Key Foundational Work

Classification with abstention but without disparities (2021) — contributed 4 concept elements to this direction.
Fair Selective Classification Via Sufficiency (2021) — contributed 3 concept elements to this direction.
Selective Classification Can Magnify Disparities Across Groups (2021) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_selective_classification_bias_aacc0ccb,
  title = {selective classification bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-classification-bias/rdr_aacc0ccb552b421ebfca7159ef400963},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_selective_classification_bias_aacc0ccb_v1,
  title = {selective classification bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-classification-bias/rdr_aacc0ccb552b421ebfca7159ef400963/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake