Diagnosing and Mitigating Bias via Feature Attribution

117 concepts80 papers1 proposal

Traditional algorithmic fairness metrics act like a smoke detector: they tell you if a fire exists (such as a demographic disparity in model predictions), but they fail to pinpoint where the fire started or how to extinguish it. As noted by Peter W Chang et al. in Feature Importance Disparities for Data Bias Investigations, high-level disparity scores offer little guidance on whether a developer should collect more data, re-engineer features, or apply algorithmic constraints.

To bridge this gap, a powerful paradigm has emerged that leverages Explainable AI (XAI) and feature attribution as both diagnostic and prescriptive tools. Instead of treating fairness and interpretability as separate goals, this approach uses feature importance to uncover hidden proxy variables, audit decision pathways, and directly modify model training objectives to enforce equitable outcomes.

Why Outcome Fairness is Blind to Process Bias

A model can achieve perfect demographic parity on paper while still relying on deeply discriminatory logic. If a model ignores a protected attribute (fairness as unawareness) but heavily weights a proxy feature—such as zip code for race, or specific non-sensitive identifiers in ranking tasks—it perpetuates the same systemic bias.

To detect this, Carlos Mougan et al. introduce the concept of "Explanation Disparity" in Measuring the Impact of Equal Treatment as Blindness via Distributions of Explanations Disparity. The intuition is straightforward: if a model treats demographic groups equally, the features driving its decisions should have similar attribution profiles across those groups. By analyzing how feature attributions (such as Shapley values) are distributed, researchers can see if a model uses entirely different reasoning pathways for different populations.

As Carlos Mougan et al. argue in Beyond Demographic Parity: Redefining Equal Treatment, evaluating the variance in feature importance across groups allows practitioners to unmask the exact variables acting as latent proxies for protected characteristics. Similarly, Peter W Chang et al. demonstrate that high levels of feature importance disparity serve as a strong proxy metric for traditional fairness violations, including increased false positive rates and poor calibration.

Pinpointing the Source of Bias with Attribution Metrics

How do we mathematically isolate a feature's discriminatory influence? One approach adapts cooperative game theory to black-box classifiers. Amit Datta et al. in Influence in Classification via Cooperative Game Theory utilize a generalization of the Banzhaf power index to measure feature influence, pinpointing exactly when a protected characteristic has a disproportionate impact on automated decisions.

In the domain of Large Language Models (LLMs), Kenza Benkirane et al. apply feature attribution mapping in How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making?. By swapping demographic attributes to create counterfactual patient variations, they isolate whether an LLM's clinical decision-making process is weighted by clinical facts or identity tokens.

For text and vision models, Mengnan Du et al. demonstrate in Data-Centric Explainable Debiasing for Improving Fairness in Pre-trained Language Models that feature attribution methods like Integrated Gradients can identify "implicit bias words"—tokens that are not explicitly demographic (like physician or tears) but carry strong, data-driven spurious associations. Furthermore, single-item explanations can sometimes mask listwise biases. To solve this in ranking systems, Maarten de Rijke introduces RankingSHAP - Faithful Listwise Feature Attribution Explanations for Ranking Models, showing that listwise feature attribution can reveal discriminatory ordering logic that pointwise explanation methods completely miss.

From Magnitude to Direction: Signed Feature Importance

Standard interpretability tools only tell you the magnitude of an input's influence—how much it matters for accuracy. They fail to tell you whether that feature is actively helping or hurting equitable outcomes.

To address this, Camille Olivia Little et al. developed FairTreeFIS in Fair Feature Importance Scores for Interpreting Decision Trees. Instead of measuring traditional impurity reduction, FairTreeFIS calculates the change in group bias metrics (such as demographic parity or equality of opportunity) at each split in a decision tree. Crucially, these scores are signed:

  • Positive scores highlight features that actively mitigate disparities and improve fairness.
  • Negative scores unmask the features driving disparate impact.

This framework was extended by Camille Little et al. in Fair Feature Importance Scores via Feature Occlusion and Permutation to create model-agnostic fair feature importance scores. By leveraging feature permutation and occlusion, this method successfully isolates features that drive unfairness from those that provide legitimate predictive signals across any model architecture.

For complex black boxes, a parallel approach developed in Fair Feature Importance Scores for Interpreting Tree-Based Methods and Surrogates trains a transparent surrogate model to approximate the black box, applying directional importance metrics to the surrogate to infer the underlying system's bias drivers.

Fairness-Aware Splitting Criteria in Tree Induction

Once feature attribution diagnostics reveal how features propagate bias, we can embed these insights directly into model construction. The most prominent application of this is in modifying the splitting criteria of decision trees and random forests.

In a standard decision tree, a node splits on the feature that maximizes information gain regarding the target label. However, Ling Luo et al. show in Discrimination-Aware Classifiers for Student Performance Prediction that this criterion can be modified to penalize information gain related to sensitive attributes. The tree is forced to select features that provide predictive power without inadvertently revealing protected group membership.

Mathematically, this regularized splitting criterion can be framed as a multi-objective optimization problem:

$$\text{Split Score} = \text{Information Gain}(\text{Target}) - \lambda \cdot \text{Information Gain}(\text{Sensitive Attribute})$$

As demonstrated in Fair Forests: Regularized Tree Induction to Minimize Model Bias, subtracting the information gain of the protected variable forces the model to bypass proxy features.

Wenbin Zhang et al. adapted this concept to streaming data environments in FAHT: An Adaptive Fairness-aware Decision Tree Classifier, evaluating statistical disparity reductions alongside traditional information gain to handle concept drift without sacrificing equity. To provide mathematical guarantees, Martin Vechev introduced FARE: Provably Fair Representation Learning with Practical Certificates, which uses a fairness-aware splitting criterion to ensure that the discrete partitions in leaf nodes retain no significant correlation with protected characteristics, yielding provable fairness certificates.

Enforcing Feature Orthogonality in Deep Networks

For deep neural networks where discrete node splitting is not possible, mitigation takes the form of regularizing the model's internal representation space or gradient pathways based on feature influence.

Mengnan Du et al. introduce this strategy in Towards Debiasing DNN Models from Spurious Feature Influence via the DeFI (Decorrelating Feature Influence) framework. DeFI trains a "bias-only" teacher model to capture sensitive features and then forces a student model to learn orthogonal feature importances. By explicitly penalizing the student model for sharing reasoning pathways with the biased teacher, the network shifts its focus from spurious shortcuts to legitimate task signals.

[ Input Data ] ───┬───> [ Bias-Only Teacher Model ] ───> [ Bias Feature Importance ]
                  │                                                    │
                  │                                           (Enforce Orthogonality)
                  │                                                    │
                  └───> [ Task-Oriented Student ]  ───> [ Clean Feature Importance ]

Similarly, Hoin Jung et al. in A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks isolate bias-prone dimensions within frozen embedding vectors of vision-language models. By training a non-linear classifier on sensitive attributes, they calculate feature importance scores for individual vector dimensions, allowing targeted debiasing without full model retraining.

By turning interpretability tools from passive observers into active constraints, these methods ensure that fairness is baked directly into the reasoning pathways of the model.

Key Foundational Work

Discrimination Aware Decision Tree Learning (2010) — contributed 2 concept elements to this direction.
Discrimination-Aware Classifiers for Student Performance Prediction (2015) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fairness_aware_splitting_criterion_64cbe69e,
  title = {fairness-aware splitting criterion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fairness-aware-splitting-criterion/rdr_64cbe69e815e47f0a23573ff8cddcc09},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fairness_aware_splitting_criterion_64cbe69e_v1,
  title = {fairness-aware splitting criterion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fairness-aware-splitting-criterion/rdr_64cbe69e815e47f0a23573ff8cddcc09/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake