When Mitigating Bias is Unfair: A Comprehensive Study on the Impact of Bias Mitigation Algorithms

Natasa Krco, Thibault Laugel, Jean-Michel Loubes, Marcin Detyniecki

Abstract

Most works on the fairness of machine learning systems focus on the blind optimization of common fairness metrics, such as Demographic Parity and Equalized Odds. In this paper, we conduct a comparative study of several bias mitigation approaches to investigate their behaviors at a fine grain, the prediction level. Our objective is to characterize the differences between fair models obtained with different approaches. With comparable performances in fairness and accuracy, are the different bias mitigation approaches impacting a similar number of individuals? Do they mitigate bias in a similar way? Do they affect the same individuals when debiasing a model? Our findings show that bias mitigation approaches differ a lot in their strategies, both in the number of impacted individuals and the populations targeted. More surprisingly, we show these results even apply for several runs of the same mitigation approach. These findings raise questions about the limitations of the current group fairness metrics, as well as the arbitrariness, hence unfairness, of the whole debiasing process.

Introduction

In the field of algorithmic fairness, practitioners typically aim for global metrics like Demographic Parity (equal acceptance rates) or Equalized Odds (equal error rates). However, these aggregate scores are often "underspecified." Multiple models can achieve identical fairness and accuracy scores on a corporate dashboard while making drastically different decisions for the same individuals. This phenomenon, known as predictive multiplicity, means that the choice of which "fair" model to deploy is often arbitrary, potentially introducing new forms of unfairness at the individual level.

Conceptual illustration of predictive multiplicity: two fair models can achieve identical global metrics by changing outcomes for entirely different sets of individuals.

The FRAME Framework

To pull back the curtain on the debiasing process, the authors introduce FRAME (FaiRness Arbitrariness and Multiplicity Evaluation). Instead of relying on a single global score, FRAME audits the transition from a biased baseline to a fair model across five dimensions:

  1. Impact Size: The percentage of the population whose predictions were changed to achieve fairness.
  2. Change Direction: Whether fairness is achieved by "leveling up" (accepting more disadvantaged individuals) or "leveling down" (rejecting more privileged individuals).
  3. Decision Rates: How the overall acceptance rate of the model shifts.
  4. Affected Subpopulations: Identifying specific types of people whose outcomes were altered using explainability tools.
  5. Neglected Subpopulations: Detecting "local" unfairness that persists within specific data slices despite global parity.

Key Findings

The Disagreement Problem

The study found that models with nearly identical aggregate scores often have a surprisingly low Intersection over Union (IOU). In some cases, the overlap between individuals targeted by different "fair" models was less than 9%. This suggests that different algorithms are "fixing" bias by targeting completely different people, making the outcome for any single individual highly dependent on the choice of algorithm.

Impact Size and Stability

The "cost" of fairness in terms of individual prediction changes varies wildly. As shown in the figure below, some methods like EGR require changing up to 31% of predictions on certain datasets, while others are far more efficient. Furthermore, non-deterministic methods (like Adversarial Debiasing) exhibit significant instability; in the Credit dataset, only 2% of the individuals targeted for a change were consistently affected across multiple training runs.

Impact size across datasets for methods optimizing Demographic Parity (left) and Equalized Odds (right), highlighting the high variability in how many individuals are affected by different algorithms.

Global vs. Local Fairness

A critical insight is that global fairness does not guarantee local fairness. A model can achieve perfect aggregate scores while remaining highly unfair to specific subgroups. For example, in the Adult dataset, most debiasing methods failed to improve fairness for individuals under 26, despite looking "fair" on the dataset as a whole.

Conclusion

The findings reveal that bias mitigation is not a neutral technical process. The arbitrariness of which individuals are affected by "fairness" interventions raises significant ethical and legal questions. Future research should prioritize "change-efficient" debiasing algorithms and develop secondary criteria to help practitioners choose between multiple fair models in a non-arbitrary, socially responsible way.

Summary updated

Abstract

Most works on the fairness of machine learning systems focus on the blind optimization of common fairness metrics, such as Demographic Parity and Equalized Odds. In this paper, we conduct a comparative study of several bias mitigation approaches to investigate their behaviors at a fine grain, the prediction level. Our objective is to characterize the differences between fair models obtained with different approaches. With comparable performances in fairness and accuracy, are the different bias mitigation approaches impacting a similar number of individuals? Do they mitigate bias in a similar way? Do they affect the same individuals when debiasing a model? Our findings show that bias mitigation approaches differ a lot in their strategies, both in the number of impacted individuals and the populations targeted. More surprisingly, we show these results even apply for several runs of the same mitigation approach. These findings raise questions about the limitations of the current group fairness metrics, as well as the arbitrariness, hence unfairness, of the whole debiasing process.
BibTeX
@inproceedings{krco2023when,
  author = {Natasa Krco and Thibault Laugel and Jean-Michel Loubes and Marcin Detyniecki},
  title = {{When Mitigating Bias is Unfair: A Comprehensive Study on the Impact of Bias Mitigation Algorithms}},
  booktitle = {SaTML},
  year = {2023},
  url = {https://openreview.net/forum?id=JzE2VhPk4H},
}
Figure