Navigating Ensemble Configurations for Algorithmic Fairness

Michael Feffer, Martin Hirzel, Samuel C. Hoffman, Kiran Kate, Parikshit Ram, Avraham ShinnararXiv 2022 · DOI · arXiv

Introduction

Algorithmic fairness is a critical yet elusive goal in machine learning. While researchers have developed numerous bias mitigators—algorithms designed to reduce discrimination against underprivileged groups—these tools often suffer from high volatility. A mitigator that achieves a high fairness score on one data split may perform poorly on another, making it difficult for practitioners to trust these models in production.

In traditional machine learning, ensembles (combining multiple models) are used to improve stability and performance. This paper explores how ensembles—such as bagging, boosting, voting, and stacking—can be systematically combined with bias mitigators to stabilize fairness gains and provides a structured framework for navigating this complex configuration space.

Methodology: The Configuration Search Space

The authors extended Lale, an open-source automated machine learning (AutoML) library, to enable the modular composition of fairness pipelines. They explored a vast search space defined by the Cartesian product of eight bias mitigators and four ensemble types. These mitigators are categorized based on where they intervene in the pipeline: pre-estimator (data transformation), in-estimator (algorithmic modification), or post-estimator (prediction adjustment).

Taxonomy of ensemble configurations for fairness mitigation.

To navigate this space, the researchers employed two primary search strategies: brute-force grid search to identify "best-in-class" configurations and Bayesian optimization (via Hyperopt) using a blended scorer that balances predictive accuracy (F1 score) and fairness (Disparate Impact).

Datasets and Experimental Rigor

The study utilized 13 datasets of varying sizes and baseline fairness levels, ranging from small benchmarks like COMPAS and Credit-g to large-scale datasets like Adult and MEPS. To ensure robust results, the authors introduced a rigorous preprocessing step using stratification on both the target label and protected attributes (e.g., race or gender), ensuring more homogeneous performance across data splits.

Key Insights: Stability and Guidance

The most significant finding is that ensembles do not just make models "fairer" in isolation; they make fairness stable. Bagging, in particular, consistently reduced the volatility of fairness metrics across different data splits. For instance, while a standalone mitigator might show high variance in its Disparate Impact score, a bagged version of that same mitigator significantly narrows the performance gap between different folds of data.

To help practitioners apply these findings, the authors distilled their results into a "guidance diagram." This flowchart directs users to specific configurations based on the size of their dataset and the severity of the baseline unfairness.

Guidance diagram for selecting fairness-mitigating ensemble configurations.

Results and Trade-offs

  • Volatility Reduction: Experimental results showed that Bagging reduced the standardized volatility of Disparate Impact from 0.18 (no ensemble) to 0.08.
  • Performance Trade-offs: While mitigation typically incurs a slight penalty in predictive performance (F1 score), mitigated ensembles maintained a superior balance between performance and stability compared to standalone mitigators.
  • Generalizability: A "leave-one-out" evaluation confirmed that the insights captured in the guidance diagram remain accurate even when specific datasets are removed from the training pool, suggesting the findings are broadly applicable.

Conclusion and Future Work

This research demonstrates that ensembles are a powerful tool for stabilizing algorithmic fairness. By providing a modular library and a practical guidance diagram, the authors offer a roadmap for practitioners to move beyond "fickle" fairness results toward more reliable, production-ready models. Future work will explore how these configurations affect a broader range of metrics, such as Equalized Odds, and investigate the impact of feature engineering on debiasing efforts.

Summary updated

Abstract

Bias mitigators can improve algorithmic fairness in machine learning models, but their effect on fairness is often not stable across data splits. A popular approach to train more stable models is ensemble learning, but unfortunately, it is unclear how to combine ensembles with mitigators to best navigate trade-offs between fairness and predictive performance. To that end, we built an open-source library enabling the modular composition of 8 mitigators, 4 ensembles, and their corresponding hyperparameters, and we empirically explored the space of configurations on 13 datasets. We distilled our insights from this exploration in the form of a guidance diagram for practitioners that we demonstrate is robust and reproducible.
BibTeX
@misc{feffer2022navigating,
  author = {Michael Feffer and Martin Hirzel and Samuel C. Hoffman and Kiran Kate and Parikshit Ram and Avraham Shinnar},
  title = {{Navigating Ensemble Configurations for Algorithmic Fairness}},
  year = {2022},
  eprint = {2210.05594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.05594},
}
Figure