The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric

Nathan Kallus, Angela Zhou

Introduction

Most fairness audits for AI risk scores focus on binary classification—evaluating whether a model correctly predicts a "yes" or "no" outcome. However, in many real-world applications, risk scores are used to rank and prioritize individuals rather than just classify them. For instance, in healthcare, a risk score might determine who receives preventative care first; in criminal justice, it might guide the level of social service intervention. This paper argues that traditional fairness metrics are insufficient for these ranking tasks and introduces xAUC (cross-AUC) to reveal deep-seated disparities that standard checks miss.

The Bipartite Ranking Framework

The authors shift the focus from classification to bipartite ranking: the task of ensuring that any given "positive" instance (e.g., someone who will recidivate) is ranked higher than any "negative" instance (e.g., someone who will not). To measure fairness in this context, they introduce two novel tools:

  1. xROC (Cross-ROC) Curve: A variation of the standard Receiver Operating Characteristic curve that plots the True Positive Rate of one group against the False Positive Rate of another.
  2. xAUC (Cross-Area Under the Curve): This metric calculates the probability that a randomly chosen "positive" member of group A is ranked higher than a randomly chosen "negative" member of group B.

By comparing $xAUC(A, B)$ with $xAUC(B, A)$, researchers can identify xAUC disparity. If these values differ significantly, the model is better at distinguishing "guilty" members of one group from "innocent" members of another than it is for the reverse pairing.

KDE distributions illustrating the overlap between cross-group pairs, showing how score distributions for innocents of one group can overlap with recidivists of another.

The "Accuracy Equity" Illusion

A central finding of the paper is that a model can exhibit "accuracy equity"—equal within-group AUCs—while still being fundamentally unfair in cross-group rankings. A model might be equally good at ranking Black defendants against other Black defendants and White defendants against other White defendants, yet fail miserably when ranking them against each other. This insight helps reconcile the famous debate over the COMPAS recidivism tool: while the model's defenders claimed accuracy equity, the authors show that the cross-group disparity was massive.

Key Results: Auditing COMPAS and Adult Income

The most striking results came from the COMPAS and Adult datasets:

  • COMPAS: The model was slightly more accurate at ranking Black defendants within their own group ($AUC = 0.737$) than White defendants ($AUC = 0.701$). However, the xAUC revealed a severe inequity: Black "innocents" (non-recidivists) were nearly indistinguishable from White "guilty" individuals (recidivists). The $\Delta xAUC$ gap was -0.21, a massive disparity hidden by standard metrics.
  • Adult Income: The model showed a tendency to rank low-income White individuals higher than high-income Black individuals. In a lending context, this would translate to withholding economic opportunity from high-earning Black applicants while over-extending it to lower-earning White applicants.
Comparison between standard within-group ROC curves and the proposed cross-group XROC curves for the COMPAS dataset, highlighting the disparity hidden by standard metrics.

Conclusion and Future Work

The xAUC metric serves as a vital diagnostic tool, identifying unfairness that traditional metrics overlook. While the authors demonstrate post-processing methods to equalize xAUC, they note potential conflicts with other fairness goals like calibration. The paper concludes that while xAUC identifies how a ranking is unfair, the underlying normative question of what constitutes a "perfectly fair" ranking remains a critical area for future research, particularly in integrating these constraints directly into the model training phase.

Summary updated

BibTeX
@inproceedings{kallus2019fairness,
  author = {Nathan Kallus and Angela Zhou},
  title = {{The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=sus-LhzaSmc},
}
Figure