Detection and evaluation of bias-inducing features in machine learning

Moses Openja, Gabriel Laberge, Foutse Khomh

TL;DR

This paper introduces a novel method for systematically identifying features that induce bias in machine learning models, even when those features aren't explicitly known. By "swapping" feature values and observing changes in model predictions, they can pinpoint and quantify bias, offering a crucial step towards building fairer AI systems.

What Problem Does This Paper Address?

Machine learning models are increasingly used in critical decision-making areas like healthcare, finance, and criminal justice. However, these models can inherit and even amplify biases present in the training data, leading to unfair outcomes for certain groups. Existing methods often require pre-identifying "sensitive" features, which can miss other bias-inducing factors or fail when sensitive features are unknown. This paper tackles the challenge of comprehensively detecting and evaluating all potentially bias-inducing features, enabling domain experts to make informed decisions about fairness.

Methods

The core of the proposed method is a "cause-to-effect" analysis inspired by counterfactual reasoning. They introduce two key techniques:

  • Single Feature Swapping: This involves modifying the values of a single feature while keeping others constant and observing how the model's prediction distribution changes. This helps estimate the "Controlled Direct Impact" of a feature.
  • Double Features Swapping: This extends the single swap by considering pairs of features (the target feature and its "mediating variables") to estimate the "Total Natural Impact," accounting for indirect influences.

These swapping functions are applied to test datasets, and the divergence (using metrics like Hellinger distance, Jensen-Shannon divergence, Total Variation distance, and Wasserstein distance) between the original and swapped model predictions is measured. Larger divergences indicate a greater bias-inducing impact. They also integrate SHAP (SHapley Additive exPlanations) to compare their findings with feature importance, highlighting that bias and importance are distinct concepts.

Datasets

The authors evaluated their approach on four well-known datasets:

  • Student Dataset: Predicting student grades.
  • Cleveland Heart Dataset: Predicting the presence of heart disease.
  • COMPAS Recidivism Dataset: Predicting the likelihood of a defendant reoffending.
  • Bank Dataset: Predicting whether a client will subscribe to a term deposit.

Key Insights and Ideas

  • Bias is often hidden: The study emphasizes that bias-inducing features may not always be the explicitly sensitive ones, and a systematic approach is needed to uncover them.
  • Direct vs. Indirect Bias: The distinction between "Controlled Direct Impact" and "Total Natural Impact" is crucial for understanding how features influence model decisions, both directly and through mediating variables.
  • Bias and Importance are separate: The research highlights that a feature can be highly important for model prediction but not necessarily bias-inducing, and vice-versa. Treating these as distinct concepts is vital for effective bias mitigation.
  • Domain expertise is key: The framework aims to support, not replace, domain experts in making decisions about acceptable levels of bias based on context.

Key Results

The paper demonstrates that their swapping functions can effectively identify features that introduce bias across various datasets and model types. For instance, in the Student Dataset, features like 'higher', 'Medu', and 'absences' showed a high direct impact, while 'sex' and 'activities' had a minimal direct impact but a higher impact when mediating variables were considered. The COMPAS Recidivism Dataset showed 'age' and 'sex' as having maximal statistical distances, indicating high direct impact. The study also showed that removing less important but bias-inducing features could improve fairness without significantly harming predictive performance, while removing important but bias-inducing features often required bias mitigation techniques.

Limitations and Future Work

  • Confounding Variables: The authors acknowledge that a genuine conclusion about bias requires considering confounding variables, which can be challenging. Their approach aims to address this by considering mediating variables.
  • Generalizability: While evaluated on four datasets, extending the framework to other domains like image processing and text mining is a future direction.
  • Mitigation Integration: The current work focuses on detection and evaluation. Future work will integrate bias mitigation techniques more deeply and explore which techniques work best for different types of bias.
  • Non-Probabilistic Models: Extending the framework to support non-probabilistic classification models is also suggested.

Summary updated

Abstract

The cause-to-effect analysis can help us decompose all the likely causes of a problem, such as an undesirable business situation or unintended harm to the individual(s). This implies that we can identify how the problems are inherited, rank the causes to help prioritize fixes, simplify a complex problem and visualize them. In the context of machine learning (ML), one can use cause-to-effect analysis to understand the reason for the biased behavior of the system. For example, we can examine the root causes of biases by checking each feature for a potential cause of bias in the model. To approach this, one can apply small changes to a given feature or a pair of features in the data, following some guidelines and observing how it impacts the decision made by the model (i.e., model prediction). Therefore, we can use cause-to-effect analysis to identify the potential bias-inducing features, even when these features are originally are unknown. This is important since most current methods require a pre-identification of sensitive features for bias assessment and can actually miss other relevant bias-inducing features, which is why systematic identification of such features is necessary. Moreover, it often occurs that to achieve an equitable outcome, one has to take into account sensitive features in the model decision. Therefore, it should be up to the domain experts to decide based on their knowledge of the context of a decision whether bias induced by specific features is acceptable or not. In this study, we propose an approach for systematically identifying all bias-inducing features of a model to help support the decision-making of domain experts. Our technique is based on the idea of swapping the values of the features and computing the divergences in the distribution of the model prediction using different distance functions. We evaluated our technique using four well-known datasets to showcase how our contribution can help spearhead the standard procedure when developing, testing, maintaining, and deploying fair/equitable machine learning systems.
BibTeX
@inproceedings{openja2024detection,
  author = {Moses Openja and Gabriel Laberge and Foutse Khomh},
  title = {{Detection and evaluation of bias-inducing features in machine learning}},
  booktitle = {Empir. Softw. Eng},
  year = {2024},
  url = {https://openreview.net/forum?id=QnrSHs9q2s},
}
Figure