With Malice Towards None: Assessing Uncertainty via Equalized Coverage

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, Emmanuel J. CandèsarXiv 2019 · OpenReview

Abstract

An important factor to guarantee a fair use of data-driven recommendation systems is that we should be able to communicate their uncertainty to decision makers. This can be accomplished by constructing prediction intervals, which provide an intuitive measure of the limits of predictive performance. To support equitable treatment, we force the construction of such intervals to be unbiased in the sense that their coverage must be equal across all protected groups of interest. We present an operational methodology that achieves this goal by offering rigorous distribution-free coverage guarantees holding in finite samples. Our methodology, equalized coverage, is flexible as it can be viewed as a wrapper around any predictive algorithm. We test the applicability of the proposed framework on real data, demonstrating that equalized coverage constructs unbiased prediction intervals, unlike competitive methods.

Introduction

Machine learning models are increasingly deployed in high-stakes domains such as criminal justice, healthcare, and university admissions. While these models offer powerful predictive capabilities, they often exhibit systematic biases, performing significantly better for majority populations than for protected groups.

Most existing fairness research focuses on point predictions—the single "best guess" of an outcome. However, point predictions are insufficient for ethical decision-making; they fail to communicate the model's uncertainty. If a model's 90% confidence interval covers the true outcome 95% of the time for one group but only 80% for another, the system is fundamentally biased in its communication of risk. This paper introduces "equalized coverage," a framework designed to ensure that uncertainty estimates are equally reliable across all demographic groups.

The Equalized Coverage Framework

The authors propose Group-conditional Conformalized Quantile Regression (CQR), a distribution-free "wrapper" that can be applied to any existing predictive algorithm. Unlike traditional statistical methods that assume data follows a specific distribution (e.g., Normal), this approach provides rigorous mathematical guarantees for any data distribution, provided the samples are exchangeable.

The methodology operates in three primary stages:

  1. Initial Training: A base quantile regression model is trained to estimate the upper and lower bounds of the target outcome.
  2. Calibration: Using a held-out calibration dataset, the model calculates conformity scores (errors) for each protected group (e.g., defined by race or gender).
  3. Adjustment: A group-specific correction factor is applied. If the model is over-confident for a specific group, the interval is widened; if it is too conservative, the interval is narrowed.

This process ensures that the resulting prediction intervals achieve the target coverage level (e.g., exactly 90%) for every group, even in finite-sample scenarios.

Empirical Evaluation

The framework was tested using the Medical Expenditure Panel Survey (MEPS) 2016 dataset, which contains 15,656 individuals and 139 features. The goal was to predict health care system utilization scores, with race ("white" vs. "non-white") serving as the sensitive attribute.

Figure 1: Cumulative distribution functions (CDFs) of signed residuals R = Y - Y_hat for white and non-white groups. The disparity between the curves illustrates why standard models fail to provide equal coverage without group-specific calibration.

Key Results

  • Failure of Marginal Models: Standard models that ignore group membership failed to provide equitable uncertainty. For a target coverage of 90%, these models achieved approximately 92% coverage for non-white individuals but only 87% for white individuals. This discrepancy means the model was "dishonest" about its certainty for the white group.
  • Success of Equalized Coverage: The proposed Group-conditional CQR achieved exactly 90% coverage (within statistical noise) for both groups.
  • Efficiency through Joint Training: The authors found that training a single model on the entire population while calibrating by group (the "Joint" mode) produced shorter, more informative intervals than training separate models for each group. This demonstrates the model's ability to "borrow strength" from the whole population while maintaining fairness.

Discussion and Insights

A key insight of this work is that uncertainty can be used as a tool for fairness. Instead of forcing a model to be equally accurate—which may be impossible due to inherent differences in data quality—we should force it to be equally honest about its limitations.

While the method guarantees equal coverage, it does not guarantee equal length. If data for one group is noisier, their intervals will be wider. The authors argue this is a desirable feature: it makes the limits of predictive performance explicit to human decision-makers.

Limitations and Future Work

Implementing equalized coverage requires access to protected attributes at the time of calibration and prediction, which may be restricted in certain legal contexts. Additionally, while wider intervals provide transparency, the disparity in interval length itself could be perceived as a form of unfairness. Future research will explore the relationship between equalized coverage and other fairness criteria, as well as extensions to categorical outcomes.

Summary updated

Abstract

An important factor to guarantee a fair use of data-driven recommendation systems is that we should be able to communicate their uncertainty to decision makers. This can be accomplished by constructing prediction intervals, which provide an intuitive measure of the limits of predictive performance. To support equitable treatment, we force the construction of such intervals to be unbiased in the sense that their coverage must be equal across all protected groups of interest. We present an operational methodology that achieves this goal by offering rigorous distribution-free coverage guarantees holding in finite samples. Our methodology, equalized coverage, is flexible as it can be viewed as a wrapper around any predictive algorithm. We test the applicability of the proposed framework on real data, demonstrating that equalized coverage constructs unbiased prediction intervals, unlike competitive methods.
BibTeX
@misc{romano2019with,
  author = {Yaniv Romano and Rina Foygel Barber and Chiara Sabatti and Emmanuel J. Candès},
  title = {{With Malice Towards None: Assessing Uncertainty via Equalized Coverage}},
  year = {2019},
  eprint = {1908.05428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.05428},
}
Figure