Introduction
Algorithmic fairness is a critical yet elusive goal in machine learning. While researchers have developed numerous bias mitigators—algorithms designed to reduce discrimination against underprivileged groups—these tools often suffer from high volatility. A mitigator that achieves a high fairness score on one data split may perform poorly on another, making it difficult for practitioners to trust these models in production.
In traditional machine learning, ensembles (combining multiple models) are used to improve stability and performance. This paper explores how ensembles—such as bagging, boosting, voting, and stacking—can be systematically combined with bias mitigators to stabilize fairness gains and provides a structured framework for navigating this complex configuration space.
Methodology: The Configuration Search Space
The authors extended Lale, an open-source automated machine learning (AutoML) library, to enable the modular composition of fairness pipelines. They explored a vast search space defined by the Cartesian product of eight bias mitigators and four ensemble types. These mitigators are categorized based on where they intervene in the pipeline: pre-estimator (data transformation), in-estimator (algorithmic modification), or post-estimator (prediction adjustment).
To navigate this space, the researchers employed two primary search strategies: brute-force grid search to identify "best-in-class" configurations and Bayesian optimization (via Hyperopt) using a blended scorer that balances predictive accuracy (F1 score) and fairness (Disparate Impact).
Datasets and Experimental Rigor
The study utilized 13 datasets of varying sizes and baseline fairness levels, ranging from small benchmarks like COMPAS and Credit-g to large-scale datasets like Adult and MEPS. To ensure robust results, the authors introduced a rigorous preprocessing step using stratification on both the target label and protected attributes (e.g., race or gender), ensuring more homogeneous performance across data splits.
Key Insights: Stability and Guidance
The most significant finding is that ensembles do not just make models "fairer" in isolation; they make fairness stable. Bagging, in particular, consistently reduced the volatility of fairness metrics across different data splits. For instance, while a standalone mitigator might show high variance in its Disparate Impact score, a bagged version of that same mitigator significantly narrows the performance gap between different folds of data.
To help practitioners apply these findings, the authors distilled their results into a "guidance diagram." This flowchart directs users to specific configurations based on the size of their dataset and the severity of the baseline unfairness.
Results and Trade-offs
- Volatility Reduction: Experimental results showed that Bagging reduced the standardized volatility of Disparate Impact from 0.18 (no ensemble) to 0.08.
- Performance Trade-offs: While mitigation typically incurs a slight penalty in predictive performance (F1 score), mitigated ensembles maintained a superior balance between performance and stability compared to standalone mitigators.
- Generalizability: A "leave-one-out" evaluation confirmed that the insights captured in the guidance diagram remain accurate even when specific datasets are removed from the training pool, suggesting the findings are broadly applicable.
Conclusion and Future Work
This research demonstrates that ensembles are a powerful tool for stabilizing algorithmic fairness. By providing a modular library and a practical guidance diagram, the authors offer a roadmap for practitioners to move beyond "fickle" fairness results toward more reliable, production-ready models. Future work will explore how these configurations affect a broader range of metrics, such as Equalized Odds, and investigate the impact of feature engineering on debiasing efforts.
Summary updated