TL;DR
This paper introduces a novel method for systematically identifying features that induce bias in machine learning models, even when those features aren't explicitly known. By "swapping" feature values and observing changes in model predictions, they can pinpoint and quantify bias, offering a crucial step towards building fairer AI systems.
What Problem Does This Paper Address?
Machine learning models are increasingly used in critical decision-making areas like healthcare, finance, and criminal justice. However, these models can inherit and even amplify biases present in the training data, leading to unfair outcomes for certain groups. Existing methods often require pre-identifying "sensitive" features, which can miss other bias-inducing factors or fail when sensitive features are unknown. This paper tackles the challenge of comprehensively detecting and evaluating all potentially bias-inducing features, enabling domain experts to make informed decisions about fairness.
Methods
The core of the proposed method is a "cause-to-effect" analysis inspired by counterfactual reasoning. They introduce two key techniques:
- Single Feature Swapping: This involves modifying the values of a single feature while keeping others constant and observing how the model's prediction distribution changes. This helps estimate the "Controlled Direct Impact" of a feature.
- Double Features Swapping: This extends the single swap by considering pairs of features (the target feature and its "mediating variables") to estimate the "Total Natural Impact," accounting for indirect influences.
These swapping functions are applied to test datasets, and the divergence (using metrics like Hellinger distance, Jensen-Shannon divergence, Total Variation distance, and Wasserstein distance) between the original and swapped model predictions is measured. Larger divergences indicate a greater bias-inducing impact. They also integrate SHAP (SHapley Additive exPlanations) to compare their findings with feature importance, highlighting that bias and importance are distinct concepts.
Datasets
The authors evaluated their approach on four well-known datasets:
- Student Dataset: Predicting student grades.
- Cleveland Heart Dataset: Predicting the presence of heart disease.
- COMPAS Recidivism Dataset: Predicting the likelihood of a defendant reoffending.
- Bank Dataset: Predicting whether a client will subscribe to a term deposit.
Key Insights and Ideas
- Bias is often hidden: The study emphasizes that bias-inducing features may not always be the explicitly sensitive ones, and a systematic approach is needed to uncover them.
- Direct vs. Indirect Bias: The distinction between "Controlled Direct Impact" and "Total Natural Impact" is crucial for understanding how features influence model decisions, both directly and through mediating variables.
- Bias and Importance are separate: The research highlights that a feature can be highly important for model prediction but not necessarily bias-inducing, and vice-versa. Treating these as distinct concepts is vital for effective bias mitigation.
- Domain expertise is key: The framework aims to support, not replace, domain experts in making decisions about acceptable levels of bias based on context.
Key Results
The paper demonstrates that their swapping functions can effectively identify features that introduce bias across various datasets and model types. For instance, in the Student Dataset, features like 'higher', 'Medu', and 'absences' showed a high direct impact, while 'sex' and 'activities' had a minimal direct impact but a higher impact when mediating variables were considered. The COMPAS Recidivism Dataset showed 'age' and 'sex' as having maximal statistical distances, indicating high direct impact. The study also showed that removing less important but bias-inducing features could improve fairness without significantly harming predictive performance, while removing important but bias-inducing features often required bias mitigation techniques.
Limitations and Future Work
- Confounding Variables: The authors acknowledge that a genuine conclusion about bias requires considering confounding variables, which can be challenging. Their approach aims to address this by considering mediating variables.
- Generalizability: While evaluated on four datasets, extending the framework to other domains like image processing and text mining is a future direction.
- Mitigation Integration: The current work focuses on detection and evaluation. Future work will integrate bias mitigation techniques more deeply and explore which techniques work best for different types of bias.
- Non-Probabilistic Models: Extending the framework to support non-probabilistic classification models is also suggested.
Summary updated