When Mitigating Bias is Unfair: Multiplicity and Arbitrariness in Algorithmic Group Fairness

Natasa Krco, Thibault Laugel, Vincent Grari, Jean-Michel Loubes, Marcin Detyniecki

Abstract

Most research on fair machine learning has prioritized optimizing criteria such as Demographic Parity and Equalized Odds. Despite these efforts, there remains a limited understanding of how different bias mitigation strategies affect individual predictions and whether they introduce arbitrariness into the debiasing process. This paper addresses these gaps by exploring whether models that achieve comparable fairness and accuracy metrics impact the same individuals and mitigate bias in a consistent manner. We introduce the FRAME (FaiRness Arbitrariness and Multiplicity Evaluation) framework, which evaluates bias mitigation through five dimensions: Impact Size, Change Direction, Decision Rates, Affected Subpopulations, and Neglected Subpopulations. Applying FRAME to various bias mitigation approaches across key datasets allows us to exhibit significant differences in the behaviors of debiasing methods. These findings highlight the limitations of current fairness criteria and the inherent arbitrariness in the debiasing process.

The Problem: Multiplicity and Arbitrariness

Standard fairness metrics like Demographic Parity (equal acceptance rates) or Equalized Odds (equal error rates) are often treated as "black-box" requirements. While they tell us if a model is fair according to a specific mathematical definition, they fail to explain how that fairness was achieved.

The authors argue that the process of turning a biased model into a fair one is fundamentally underspecified. A model can be made "fair" by "leveling up" (increasing opportunities for disadvantaged groups) or "leveling down" (reducing opportunities for privileged groups). From the perspective of a global score, these strategies may appear identical, but for the individuals involved, the impact is drastically different. This phenomenon, termed "multiplicity in debiasing," introduces an element of arbitrariness where different models with identical performance metrics target completely different sets of people.

Multiplicity in debiasing: Two models can achieve similar fairness and accuracy while changing predictions for different individuals.

The FRAME Framework

To expose this hidden arbitrariness, the paper introduces FRAME (FaiRness Arbitrariness and Multiplicity Evaluation). FRAME shifts the focus from final scores to the transition from a biased model ($f$) to a fair model ($g$) across five dimensions:

  1. Impact Size: The percentage of the population whose predictions were changed.
  2. Change Direction: Whether changes were positive (0 to 1) or negative (1 to 0).
  3. Decision Rates: Changes in overall acceptance rates for each group.
  4. Affected Subpopulations: Identifying features that characterize individuals targeted by different methods using Partial Least Squares Discriminant Analysis (PLS-DA).
  5. Neglected Subpopulations: Identifying local subgroups where unfairness persists despite global improvements.

Experimental Setup

The framework was evaluated on five standard fairness datasets: Adult (income), Dutch (occupation), COMPAS (recidivism), Bank (telemarketing), and Credit (loan defaults). Various bias mitigation techniques, including in-processing (e.g., Adversarial Debiasing) and post-processing (e.g., Reject Option Classification), were compared.

Key Results and Insights

The Disagreement Problem

A striking finding is the low overlap between individuals "fixed" by different debiasing algorithms. Even when two methods reach the same fairness and accuracy targets, the intersection of affected individuals is often less than 9%. This confirms that the choice of algorithm is a significant source of individual-level arbitrariness.

Efficiency vs. Overhaul

Post-processing methods tend to be more "efficient," requiring fewer prediction changes to reach fairness. In contrast, in-processing methods often overhaul the model, changing up to 31% of predictions. This high impact size can be risky for models already deployed in production environments.

The "Levelling Down" Trap and Change Direction

Bias mitigation often involves a trade-off in change direction. Some models achieve fairness primarily by rejecting individuals from the privileged group who were previously accepted—a "levelling down" strategy. As shown in the analysis of the Adult and Dutch datasets, different methods exhibit varying patterns of positive and negative changes across sensitive groups ($S=0$ and $S=1$).

Percentage of changed predictions across different methods, showing the balance of positive and negative differences for sensitive groups.

Local Unfairness

Global fairness does not guarantee local fairness. For example, in the Adult dataset, most debiasing methods failed to improve outcomes for individuals under 26, highlighting that "fair" models can still neglect specific demographic slices.

Conclusion and Future Work

FRAME provides a diagnostic lens to understand the human impact of bias mitigation. The findings suggest that current fairness criteria are insufficient to prevent arbitrary individual impacts. Future research should focus on expanding to non-binary metrics like calibration, developing "change-efficient" debiasing methods, and incorporating human-in-the-loop "secondary criteria" to guide model selection in a socially and legally responsible manner.

Summary updated

Abstract

Most research on fair machine learning has prioritized optimizing criteria such as Demographic Parity and Equalized Odds. Despite these efforts, there remains a limited understanding of how different bias mitigation strategies affect individual predictions and whether they introduce arbitrariness into the debiasing process. This paper addresses these gaps by exploring whether models that achieve comparable fairness and accuracy metrics impact the same individuals and mitigate bias in a consistent manner. We introduce the FRAME (FaiRness Arbitrariness and Multiplicity Evaluation) framework, which evaluates bias mitigation through five dimensions: Impact Size (how many people were affected), Change Direction (positive versus negative changes), Decision Rates (impact on models' acceptance rates), Affected Subpopulations (who was affected), and Neglected Subpopulations (where unfairness persists). This framework is intended to help practitioners understand the impacts of debiasing processes and make better-informed decisions regarding model selection. Applying FRAME to various bias mitigation approaches across key datasets allows us to exhibit significant differences in the behaviors of debiasing methods. These findings highlight the limitations of current fairness criteria and the inherent arbitrariness in the debiasing process.
BibTeX
@inproceedings{krco2023when,
  author = {Natasa Krco and Thibault Laugel and Vincent Grari and Jean-Michel Loubes and Marcin Detyniecki},
  title = {{When Mitigating Bias is Unfair: Multiplicity and Arbitrariness in Algorithmic Group Fairness}},
  booktitle = {SaTML},
  year = {2023},
  url = {https://openreview.net/forum?id=JzE2VhPk4H},
}
Figure