Fairness in Explainable AI Narratives: Decomposing Social Disparities in LLM-Generated Explanations

Wannes Van den Bulck, Sofie Goethals, David MartensarXiv 2026 · arXiv

TL;DR

As we move from technical AI explanations (like SHAP) to LLM-generated "narratives" for lay users, we introduce a new layer of social bias. This paper demonstrates that LLMs often prioritize narrative fluency and stereotypes over factual accuracy, creating significant disparities in how different social groups are explained—even when the underlying predictive model is identical.

What Problem Does This Paper Address?

Explainable AI (XAI) has a major "interpretability gap." While methods like SHAP provide precise feature importance scores, these technical artifacts are often incomprehensible to the people they affect most—loan applicants, students, or job seekers. To solve this, researchers have turned to Large Language Models (LLMs) to translate these scores into eXplainable AI Narratives (XAINs): structured, story-like explanations.

Fig. 1. From SHAP to XAIN: the top-3 SHAP features of a German Credit applicant (instance 18) and (part of) the narrative generated by claude-sonnet-4-6. Colours link each feature to its description in the narrative.

However, LLMs are not neutral translators. They bring their own internal biases. This paper addresses the unstudied risk of fairness in XAINs. If an LLM generates a less faithful or more negative explanation for a marginalized group, it undermines the user's right to a "clear and meaningful explanation" (as mandated by the EU AI Act) and their ability to contest adverse decisions.

Methods

The authors propose a framework to audit and "decompose" these disparities by generating narratives under three distinct conditions:

  1. Factual PA disclosed: The prompt includes the subject's true protected attribute (e.g., gender, race).
  2. Factual PA withheld: The protected attribute is removed from the prompt.
  3. Counterfactual PA disclosed: The protected attribute is swapped (e.g., changing "Male" to "Female") while keeping all other features identical.

By comparing these conditions, the researchers can pinpoint whether a disparity is caused by the feature values (indirect bias), the disclosure of the attribute itself, or the value of that attribute (direct bias).

Fig. 3. Illustration of the three narrative generation conditions and the five resulting disparity comparisons, based on real (top) and counterfactual PA-swapped (bottom) credit-dataset instances for female (F) and male (M) applicants.

The evaluation pipeline uses a separate "Extractor LLM" to turn the generated narratives back into structured data, which is then compared against the ground-truth SHAP values for faithfulness (accuracy of values, signs, and ranks), feature coverage, and sentiment.

Datasets

The framework was evaluated on four publicly available datasets representing high-risk domains defined by the EU AI Act:

  • German Credit: Loan applications (Credit Scoring).
  • Saudi HR: Employee turnover/promotion (Employment).
  • Student Performance: Secondary school performance (Education).
  • Law School Bar Passage: Bar exam success (Education).
Overview of the datasets and protected attribute splits.

Key Insights and Ideas

  • The "Plausible Falsehood": LLMs frequently "correct" counterintuitive SHAP signs to match their internal expectations. For example, if a model says "low work experience" actually helped an applicant, a smaller LLM might flip the sign in the narrative, claiming it was a risk because it sounds more "plausible."
  • Faithfulness is Driven by Features: Disparities in faithfulness (how accurately the LLM reflects the SHAP data) are mostly triggered by specific feature values that are unequally distributed across groups. The LLM makes errors that are "blind" to the group but "sensitive" to the data profile.
  • Coverage and Sentiment are Driven by Labels: Once a protected attribute like race or gender is disclosed, LLMs change how they talk. For instance, in the Law dataset, Claude 3.5 Sonnet added non-discrimination disclaimers significantly more often for non-white applicants. While well-intentioned, these disclaimers took up space, leading to less actual explanation of the features that drove the decision.
  • Fairness can be "Bought": Faithfulness strongly correlates with model cost. Frontier models like GPT-4 and Claude 3.5 Sonnet were highly faithful, while smaller, cheaper models (like GPT-4o-mini or Ministral 14b) were much more prone to "hallucinating" justifications for adverse decisions.

Key Results

  • Faithfulness Disparities: In the Saudi HR dataset, GPT-4o-mini correctly identified the sign of the most important feature for 70% of men but only 37% of women. This was because the LLM's prior expectations about "experience" overrode the actual data.
  • Sentiment Shifts: In the Student Performance dataset, GPT-4.1 generated significantly less positive narratives for boys than for girls (0.37 vs 0.44 sentiment score), a disparity that vanished when gender was withheld.
  • The "Unawareness" Trade-off: Withholding protected attributes (fairness through unawareness) removed most sentiment and coverage disparities but did nothing to fix faithfulness errors. In some cases, withholding the attribute actually caused models to fabricate more values to fill the narrative gap.

Limitations and Future Work

The authors acknowledge several limitations:

  • Language and Culture: The study was limited to English-language narratives and a single prompt template.
  • Binary Splits: Social groups were compared as binary splits (e.g., white vs. non-white), ignoring intersectional complexities.
  • Human Perception: The study measured algorithmic metrics but did not test how actual human "lay users" perceive or trust these different narrative styles.
  • Privacy vs. Quality: There is an inherent trade-off; using high-quality frontier models for better faithfulness often requires sending sensitive applicant data to third-party providers, raising significant privacy concerns.

Future work should explore extending this framework to other XAI methods beyond SHAP and investigating how these narrative disparities affect real-world user behavior and trust.

Summary updated

Abstract

Large language models (LLMs) are increasingly used to translate technical explainable AI (XAI) artefacts such as SHAP into explanatory narratives (XAINs) for lay users. In high-risk domains, these explanations are often the main resource through which individuals can understand and contest adverse decisions. Yet LLMs are known to reproduce social bias, even from proxy variables alone, and fairness in XAINs remains unstudied. Because XAINs are bound to a ground-truth artefact, fairness concerns not only what a narrative says and how, but also whether it is equally faithful across social groups. We introduce fairness in XAINs as a new field of research and propose a framework that generates narratives under three conditions (protected attribute disclosed, withheld, or counterfactually swapped), allowing five comparisons that pinpoint the cause of disparities. We apply it to adversely classified instances from four datasets in credit scoring, education and employment, generating narratives with five LLMs, and evaluate faithfulness, feature coverage and sentiment. We find significant disparities across all datasets, metric families and models, arising in two ways. Faithfulness disparities result from feature values: LLMs make plausible but false statements, triggered by values that are unequally distributed across social groups. Feature coverage and sentiment disparities stem from the protected attribute itself: once it is disclosed, LLMs give less explanation to, or frame more negatively the narratives of, groups that have historically been discriminated against or are stereotypically associated with worse outcomes. A more faithful model removes the faithfulness disparities, and withholding protected attributes mostly removes the others, but neither suffices on its own. Fairness in XAINs should therefore be audited separately from the predictive model and its XAI artefact.
BibTeX
@misc{bulck2026fairness,
  author = {Wannes Van den Bulck and Sofie Goethals and David Martens},
  title = {{Fairness in Explainable AI Narratives: Decomposing Social Disparities in LLM-Generated Explanations}},
  year = {2026},
  eprint = {2611.12592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12592},
}
Figure