The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the XAUC Metric

Nathan Kallus, Angela Zhou

TL;DR

Risk scores in criminal justice, medicine, and lending are rarely used as simple binary decision rules; instead, they rank individuals to allocate limited resources or prioritize interventions. This paper introduces xAUC (cross-AUC), an auditing metric that measures the probability that a positive individual from one protected group is ranked above a negative individual from another. Evaluating models with xAUC exposes severe cross-group ranking disparities that remain completely hidden by traditional within-group AUC and calibration metrics.

What Problem Does This Paper Address?

Algorithmic fairness research often focuses on binary classification metrics (e.g., false positive and false negative parity at a single threshold) or calibration (e.g., ensuring a 70% risk score means a 70% chance of recidivism for all groups). However, real-world decision-makers routinely use risk scores as continuous rankings—for triage, risk-based pricing, or bail supervision intensity.

In bipartite ranking, the goal is to score positive instances higher than negative instances across the board. When commercial risk assessment vendors (such as Equivant defending COMPAS) claim their tools exhibit "accuracy equity" because within-group AUC is balanced, they miss a critical flaw: a model can achieve identical or higher AUC within each group while systematically ranking non-deserving individuals from an advantaged group above deserving individuals from a disadvantaged group.

The paper fills this gap by formalizing a framework to audit ranking fairness across groups without committing to an arbitrary classification threshold.

Methods

The authors propose cross-group evaluation metrics grounded in bipartite ranking theory:

  • Cross-ROC (xROC): A curve plotting the True Positive Rate (TPR) of group $a$ against the False Positive Rate (FPR) of group $b$ across all possible classification thresholds $\theta$: $$\text{xROC}(\theta; R, a, b) = (\Pr[R > \theta \mid A = b, Y = 0],\; \Pr[R > \theta \mid A = a, Y = 1])$$
  • Cross-AUC (xAUC): The area under the xROC curve, representing the probability that a randomly chosen positive example from group $a$ scores higher than a randomly chosen negative example from group $b$: $$\text{xAUC}(a, b) = \Pr[R_1^a > R_0^b]$$
  • xAUC Disparity ($\Delta\text{xAUC}$): The gap between reciprocal cross-rankings: $$\Delta\text{xAUC} = \Pr[R_1^a > R_0^b] - \Pr[R_1^b > R_0^a]$$
  • AUC Decomposition: The authors show that standard population AUC decomposes neatly into a weighted sum of within-group AUCs and between-group xAUC metrics weighted by outcome-conditional group shares.
  • Conditional xAUC Diagnostics: Evaluating the distribution of individual conditional rank accuracies ($\Pr[R_1^a > R_0^b \mid R_0^b]$) highlights tail behavior and pinpoint whether extreme misranking concentrates among specific individuals.

Datasets

The authors evaluate models (Logistic Regression and calibrated Bipartite RankBoost) across four standard fairness benchmarks:

  1. COMPAS Recidivism Dataset ($n = 6{,}167, p = 402$): Sensitive attribute is race (Black vs. White); outcome is 2-year violent non-recidivism ($Y=1$).
  2. Adult Income Dataset ($n = 30{,}162, p = 98$): Sensitive attribute is race (Black vs. White); outcome is high income $> \$50\text{k}$ ($Y=1$).
  3. Framingham Heart Study ($n = 4{,}658, p = 7$): Sensitive attribute is gender (Male/Non-Female vs. Female); outcome is 10-year coronary heart disease incidence ($Y=1$).
  4. German Credit Dataset ($n = 1{,}000, p = 57$): Sensitive attribute is age ($< 25$ vs. $\ge 25$); outcome is non-default ($Y=1$).

Key Insights and Ideas

  • Within-group parity masks cross-group injustice: On COMPAS, the within-group AUC for Black defendants is actually higher than for White defendants (0.737 vs. 0.701). Yet cross-ranking reveals a massive gap: Black non-recidivators have only a 0.604 probability of scoring higher than White recidivators, whereas White non-recidivators have a 0.813 probability of scoring higher than Black recidivators ($\Delta\text{xAUC} = -0.21$). White recidivators effectively receive an unearned "benefit of the doubt."
  • Direct ranking optimization does not prevent disparity: Using algorithms explicitly designed for ranking (like RankBoost) yields the same large cross-group disparities as standard logistic regression, demonstrating that optimizing aggregate ranking loss does not naturally distribute ranking errors equitably.
  • Impact of score variance: Assuming normal score distributions, the authors prove in closed form that disparate variance (prediction precision) alone produces significant xAUC disparity—even if mean scores across groups are completely equal.

Key Results

  • COMPAS: $\text{xAUC}(\text{Black}, \text{White}) = 0.604$ vs. $\text{xAUC}(\text{White}, \text{Black}) = 0.813$. The conditional xAUC analysis showed that 15 White violent recidivators were ranked above virtually all Black non-recidivators.
  • Adult Dataset: Within-group AUCs are high and balanced (0.923 for Black, 0.898 for White). However, $\text{xAUC}(\text{Black}, \text{White}) = 0.865$ vs. $\text{xAUC}(\text{White}, \text{Black}) = 0.944$, showing a systematic bias where lower-income White individuals are scored above higher-income Black individuals.
  • Framingham & German Credit: Framing cross-comparisons exposed persistent ranking asymmetries, such as disparities in allocating preventive cardiac care across genders ($\text{xAUC} = 0.795$ vs. $0.737$).

Limitations and Future Work

  • Normative assumption of base-rate equipoise: xAUC conditions on draws from $Y=0$ and $Y=1$ distributions with equal weight. When true base rates differ significantly between groups, interpreting xAUC disparities as unfair assumes it is normatively desirable for random individuals from each class to face identical pairwise misranking risks.
  • Remediation and post-processing: The paper primarily focuses on audit and diagnosis. While the authors touch on post-processing adjustments in the appendix, they caution that fair ranking adjustments (such as stochastic ranking or threshold adjustments) are difficult to implement in practice without running into well-known fairness impossibility trade-offs or legal restrictions against disparate treatment.

Summary updated

Abstract

Where machine-learned predictive risk scores inform high-stakes decisions, such as bail and sentencing in criminal justice, fairness has been a serious concern. Recent work has characterized the disparate impact that such risk scores can have when used for a binary classification task. This may not account, however, for the more diverse downstream uses of risk scores and their non-binary nature. To better account for this, in this paper, we investigate the fairness of predictive risk scores from the point of view of a bipartite ranking task, where one seeks to rank positive examples higher than negative ones. We introduce the xAUC disparity as a metric to assess the disparate impact of risk scores and define it as the difference in the probabilities of ranking a random positive example from one protected group above a negative one from another group and vice versa. We provide a decomposition of bipartite ranking loss into components that involve the discrepancy and components that involve pure predictive ability within each group. We use xAUC analysis to audit predictive risk scores for recidivism prediction, income prediction, and cardiac arrest prediction, where it describes disparities that are not evident from simply comparing within-group predictive performance.
BibTeX
@inproceedings{kallus2019fairness,
  author = {Nathan Kallus and Angela Zhou},
  title = {{The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the XAUC Metric}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=sus-LhzaSmc},
}

Loading related content...

Figure