An Empirical Study of Modular Bias Mitigators and Ensembles

Michael Feffer, Martin Hirzel, Samuel C. Hoffman, Kiran Kate, Parikshit Ram, Avraham ShinnararXiv 2022 · arXiv

Introduction

Algorithmic bias is a well-documented risk in machine learning, potentially leading to discriminatory outcomes in critical sectors such as hiring and healthcare. While numerous "bias mitigators" have been developed to address these issues, they are often plagued by high volatility—a mitigator may ensure fairness on one training set but fail on another. This study investigates whether ensemble learning, a standard technique for improving model stability, can be effectively combined with bias mitigators to make algorithmic fairness more reliable.

Methodology: A Modular Framework

The researchers advocate for a modular approach to fairness. By developing an open-source library (extending IBM’s Lale and AIF360), they treat mitigators and ensembles as independent building blocks. This framework allows for the systematic exploration of three categories of mitigators:

  1. Pre-estimator: Transforms input data to remove bias prior to training.
  2. In-estimator: Integrates fairness constraints directly into the learning algorithm.
  3. Post-estimator: Adjusts the predictions of a trained model to ensure equitable output.

Ensemble Strategies

The study evaluated four primary ensemble methods to determine their impact on fairness stability:

Overview of the ensemble algorithms used in the experiments, categorized by their training approach and composition.

Composing Mitigators and Ensembles

A central question of the research is the optimal level at which to apply mitigation. Mitigation can occur at the estimator-level (applied to each individual model within the ensemble) or the ensemble-level (applied to the final aggregated output). This creates a vast search space of potential configurations.

This diagram illustrates the modular search space of possible combinations between different types of ensembles and mitigation levels.

Experimental Setup

The framework was tested across 13 diverse datasets from OpenML, ranging from small datasets like Ricci (118 rows) to large benchmarks like Adult (48,842 rows). This variety allowed the authors to observe how data volume and baseline bias levels influence the effectiveness of various interventions.

Key Insights and Results

1. Stability as the Primary Gain

The most significant finding is that ensembles effectively reduce the "fickleness" (volatility) of fairness metrics. By lowering the standard deviation of metrics such as Disparate Impact (DI) across different data splits, ensembles provide practitioners with greater confidence that fairness measured during testing will persist in production.

2. The Advantage of Estimator-Level Mitigation

The results indicate that estimator-level mitigation generally provides better stability than ensemble-level mitigation. While ensemble-level approaches are computationally more efficient (requiring only a single mitigation step), they are less robust against data variability.

3. Persistent Trade-offs

Despite the stability gains, a fundamental trade-off between fairness and accuracy (e.g., F1 score) remains. Ensembles stabilize both metrics but do not eliminate the inherent tension between these competing objectives.

4. Risks in Stacking

Stacking—using a meta-model to aggregate predictions—presents unique risks. If the final aggregator is not itself mitigated, it can inadvertently re-introduce bias that the base models had successfully removed.

Conclusion

This empirical study demonstrates that modular ensembles are a powerful tool for enhancing the reliability of algorithmic fairness. By reducing the volatility of bias mitigators, ensembles bridge the gap between experimental results and dependable real-world deployment. Future research into "AutoML for Fairness" could further streamline the process of identifying optimal mitigator-ensemble configurations for specific datasets.

Summary updated

Abstract

There are several bias mitigators that can reduce algorithmic bias in machine learning models but, unfortunately, the effect of mitigators on fairness is often not stable when measured across different data splits. A popular approach to train more stable models is ensemble learning. Ensembles, such as bagging, boosting, voting, or stacking, have been successful at making predictive performance more stable. One might therefore ask whether we can combine the advantages of bias mitigators and ensembles? To explore this question, we first need bias mitigators and ensembles to work together. We built an open-source library enabling the modular composition of 10 mitigators, 4 ensembles, and their corresponding hyperparameters. Based on this library, we empirically explored the space of combinations on 13 datasets, including datasets commonly used in fairness literature plus datasets newly curated by our library. Furthermore, we distilled the results into a guidance diagram for practitioners. We hope this paper will contribute towards improving stability in bias mitigation.
BibTeX
@misc{feffer2022empirical,
  author = {Michael Feffer and Martin Hirzel and Samuel C. Hoffman and Kiran Kate and Parikshit Ram and Avraham Shinnar},
  title = {{An Empirical Study of Modular Bias Mitigators and Ensembles}},
  year = {2022},
  eprint = {2202.00751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.00751},
}
Figure