Effectiveness of Equalized Odds for Fair Classification under Imperfect Group Information

Pranjal Awasthi, Matthäus Kleindessner, Jamie Morgenstern

Abstract

Most approaches aiming to ensure a model's fairness with respect to a protected attribute (such as gender or race) assume to know the true value of the attribute for every data point. In this paper, we ask to what extent fairness interventions can be effective even when only imperfect information about the protected attribute is available. In particular, we study the prominent equalized odds postprocessing method of Hardt et al. (2016) under a perturbation of the attribute. We identify conditions on the perturbation that guarantee that the bias of a classifier is reduced even by running equalized odds with the perturbed attribute. We also study the error of the resulting classifier. We empirically observe that under our identified conditions most often the error does not suffer from a perturbation of the protected attribute. For a special case, we formally prove this observation to be true.

Introduction

Machine learning models are increasingly used in high-stakes decision-making. To ensure these models do not perpetuate biases against protected groups (e.g., based on race or gender), researchers often apply fairness constraints. One prominent framework is Equalized Odds (EO), which requires that a model’s error rates—specifically false positives and false negatives—be equal across different groups.

However, a significant practical challenge exists: most fairness algorithms assume perfect knowledge of the protected attribute ($A$) for every individual in the training set. In real-world scenarios, group information is frequently missing, self-reported with errors, or derived from noisy proxies. This paper addresses whether EO postprocessing remains effective when group data is corrupted, providing formal guarantees for its robustness.

Methodology

The authors analyze the EO postprocessing method under a noise model where the training phase uses a corrupted attribute ($A_c$), while the test phase has access to the true attribute ($A$). They identify Assumptions I as necessary conditions for "well-behaved" noise:

  1. Conditional Independence: The noise in $A_c$ must be independent of the model's original predictions ($\tilde{Y}$), given the true label and true attribute.
  2. Moderate Perturbation: The probability of flipping the attribute must not be so high that groups become indistinguishable (specifically, the sum of flip probabilities must be $\leq 1$).

Using linear programming theory, the authors prove that if these conditions are met, the resulting "fair" classifier is guaranteed to have lower bias than the original "unfair" model.

Key Results and Insights

The paper establishes two primary theoretical results regarding the behavior of the corrected classifier ($Y_{corr}$):

  • Theorem 1 (Bias Reduction): Under the identified noise conditions, the bias of the corrected classifier is strictly less than the bias of the original classifier.
  • Theorem 2 (Error Bound): In balanced distributions, the error of the classifier derived from noisy information is lower than or equal to the error of a classifier derived from true information.
Bias and error as a function of noise level across different data distributions. The dashed lines represent the noisy classifier, while solid lines represent the baseline.

As shown in Figure 5, noise in the training labels often acts as a "softener." Because noise makes groups appear more similar than they actually are, the EO algorithm applies a less aggressive correction. This leads to an "accuracy sweet spot" where the resulting classifier retains more of its original accuracy compared to a model trained with perfect group information.

Empirical Validation

The theoretical findings were validated using synthetic simulations and three real-world datasets: the Drug Consumption Dataset, COMPAS (recidivism prediction), and the Adult Dataset (census income prediction).

On the COMPAS and Adult datasets, bias was consistently reduced as long as the noise level stayed below 0.5. Furthermore, the "cost" of fairness (accuracy loss) was often lower when group information was slightly noisy, confirming the theoretical "interpolation" effect where noise prevents over-correction.

Limitations and Future Work

While this work provides a strong foundation for robust fairness, several avenues for future research remain. The current analysis assumes the attribute can be accurately verified during the test (deployment) phase. Additionally, the study is limited to binary classification and binary protected attributes. Extending these guarantees to multi-class problems, continuous attributes, and highly unbalanced real-world scenarios remains an open challenge.

Summary updated

Abstract

Most approaches aiming to ensure a model's fairness with respect to a protected attribute (such as gender or race) assume to know the true value of the attribute for every data point. In this paper, we ask to what extent fairness interventions can be effective even when only imperfect information about the protected attribute is available. In particular, we study the prominent equalized odds postprocessing method of Hardt et al. (2016) under a perturbation of the attribute. We identify conditions on the perturbation that guarantee that the bias of a classifier is reduced even by running equalized odds with the perturbed attribute. We also study the error of the resulting classifier. We empirically observe that under our identified conditions most often the error does not suffer from a perturbation of the protected attribute. For a special case, we formally prove this observation to be true.
BibTeX
@inproceedings{awasthi2020effectiveness,
  author = {Pranjal Awasthi and Matthäus Kleindessner and Jamie Morgenstern},
  title = {{Effectiveness of Equalized Odds for Fair Classification under Imperfect Group Information}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=V-s7C7stlbJ},
}
Figure