Disparities in Machine Learning Explanation Quality

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 217 papers40 conceptsfrozen 2026-10-04

In the effort to make artificial intelligence more accountable, researchers have developed post-hoc explanation methods like SHAP and LIME. These tools are designed to pull back the curtain on "black box" models, telling us which features—like income or age—most influenced a specific decision.

However, a growing body of research suggests that these curtains do not pull back equally for everyone. While we often audit models for outcome fairness (whether the predictions are biased), a new frontier of research focuses on procedural fairness: whether the explanations themselves are of equal quality across different demographic groups.

The Gap in Explanation Fidelity

The core of this issue lies in faithfulness (or fidelity)—the degree to which an explanation accurately reflects the underlying model's true decision-making logic. Jessica Dai, Sohini Upadhyay, and Himabindu Lakkaraju have demonstrated that post-hoc explanations often exhibit significant "quality disparities." In their work, Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations, they find that these methods are frequently less stable and less accurate for minority subgroups.

To understand why this happens, consider how post-hoc explanations work: they usually approximate a complex model with a simpler, interpretable one (a "surrogate") in a local neighborhood. If a model's decision boundary is more "jagged" or complex for one demographic than another, the surrogate will struggle to capture it accurately. Zixi Chen and Finale Doshi-Velez define this consistency as homogeneity. When homogeneity is low, the "right to an explanation" is distributed unequally, leaving some populations with justifications that are essentially noise.

Fairwashing and Hidden Biases

The danger of low-fidelity explanations is not just a lack of clarity; it is the risk of fairwashing. If an explanation method fails to accurately reflect a model's reliance on sensitive attributes for a specific group, it can obscure underlying discrimination.

As Jessica Dai notes, an unfaithful explanation might suggest a model is using "neutral" features to make a decision when it is actually relying on protected characteristics. This creates a secondary layer of inequality: not only is the model potentially biased, but the tools we use to audit that bias are also failing the very people they are meant to protect.

This phenomenon extends to Large Language Models (LLMs). Katie Matton and Emre Kiciman found in Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations that LLMs are often "dishonest" in their self-explanations, systematically hiding the influence of social biases while providing plausible-sounding justifications.

Measuring Procedural Unfairness

To address these disparities, researchers are moving toward standardized, quantitative benchmarking. Chirag Agarwal and colleagues introduced OpenXAI, an ecosystem that includes metrics specifically designed to detect explanation inequality. These metrics often involve "perturbation tests": if you remove the features that an explanation claims are important, the model's prediction should change significantly. If the prediction stays the same for a certain subgroup, the explanation was unfaithful.

Yu Wang and Tyler Derr argue that we must optimize for both outcomes and explanations simultaneously. In Fairness and Explainability: Bridging the Gap towards Fair Model Explanations, they propose the Comprehensive Fairness Algorithm (CFA). This approach treats explanation quality as a first-class citizen during model training, ensuring that the model's reasoning process is as grounded and reliable for minority groups as it is for the majority.

The Limits of Personalization

A common intuition is that "personalizing" models by including demographic data will make them more accurate and transparent. However, Haewon Jeong and Nina Miolane prove that this is not always the case. In When Machine Learning Gets Personal, they use formal divergence analysis to show that improvements in predictive accuracy can actually decouple from explanation faithfulness. A model might become more "accurate" for a group by picking up on subtle, complex correlations that are then much harder for post-hoc tools to explain faithfully.

Ultimately, this research direction shifts the focus of AI ethics from the what to the how. As Christoph Kern suggests, true accountability in automated systems requires that the transparency and oversight benefits of XAI are distributed equitably across all populations. Without addressing explanation quality disparities, our "transparent" systems may remain opaque to those who need clarity the most.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_fidelity_disparity_e2cd86fa,
  title = {explanation fidelity disparity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-fidelity-disparity/rdr_e2cd86faf3504c40a4ce0e17cd96c2f3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_disparities_in_machine_learning_explanat_e2cd86fa_v1,
  title = {Disparities in Machine Learning Explanation Quality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disparities-in-machine-learning-explanation-quality/rdr_e2cd86faf3504c40a4ce0e17cd96c2f3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}