Subgroup Fidelity Disparities in Post Hoc Explanations

28 concepts12 papers1 proposal

Imagine a bank that uses a complex machine learning model to approve or deny loans. To ensure compliance and ethical standards, the bank audits the model for predictive fairness and finds that it approves loans at equitable rates across different demographic groups. To provide transparency to its customers, the bank also deploys a post-hoc explanation tool like LIME or SHAP. This tool boasts a 95% average "faithfulness"—meaning that 95% of the time, its simple, human-readable explanations accurately mirror the true internal logic of the complex model.

But average metrics can be deeply misleading. If you break down that 95% average by demographic subgroups, you might discover that the explanation tool is 99% accurate for the majority group, but only 60% accurate for a minority group.

This phenomenon is known as an explanation fidelity disparity. When it occurs, the transparency and accountability benefits of explainable AI are distributed unequally, creating a "transparency tax" where marginalized populations receive less reliable justifications for automated decisions.

The Decoupling of Prediction and Explanation

A common assumption in algorithmic fairness is that making a model's predictions fair will inherently make its explanations fair. However, a growing body of research demonstrates a fundamental decoupling between predictive fairness and explanatory fairness.

As Balagopalan et al. and Dhaini et al. show, significant fidelity gaps can persist even when the underlying predictive model has been completely debiased and achieves near-zero disparity in its outputs. This occurs because post-hoc explainers are entirely separate surrogate models trained to approximate the black box. The optimization process for these surrogates typically maximizes aggregate faithfulness across the entire dataset. Just as standard machine learning models can neglect minority subgroups to minimize global loss, post-hoc explainers sacrifice explanation quality on minority subgroups to maximize global faithfulness (Agarwal et al.).

Why Do Explanations Fail Unequally?

To understand why explanation quality degrades for specific groups, we must look at the mechanics of how post-hoc explainers operate. Many popular explainers rely on a perturbation-based framework: they take a specific data point, generate a cloud of slightly altered variations around it, pass those variations through the black-box model, and train a simple local linear model on the responses to see which features matter most.

Mhasawade et al. demonstrate that these local approximations are highly sensitive to underlying data properties, such as distribution shifts and omitted variables. If a minority subgroup lies in a sparse or highly non-linear region of the feature space, the perturbed samples generated by the explainer may fall into regions the model handles inconsistently. Consequently, the local linear surrogate fails to capture a stable representation of the model's behavior. This explains why Dhaini et al. found that perturbation-based interpretability frameworks exhibit significant subgroup quality gaps far more frequently than gradient-based approaches, which inspect the model's internal weights directly rather than sampling the surrounding space.

Furthermore, the underlying model itself may be employing entirely different reasoning processes for different groups. Wang et al. introduce the concept of "procedure-oriented fairness," pointing out that a model might rely on robust, logical features for one demographic group while exploiting noisy proxy variables for another. When a model's internal logic is inherently more fractured or diffuse for a specific population—a phenomenon Klingenberg et al. observed in medical imaging due to structural biological variability across sexes—the explainer's feature maps become scattered and unfaithful, signaling an underlying inequality in how the model processes different populations.

The Threat of Fairwashing

Fidelity disparities are not just passive technical failures; they create severe security and auditing vulnerabilities. The most prominent of these is fairwashing—the deceptive practice of using a fair-looking explanation to mask a biased black-box model.

Because post-hoc explainers struggle to maintain uniform fidelity, an adversary can exploit this slack. Aïvodji et al. demonstrate that within the set of all possible surrogate models that achieve high aggregate fidelity to a black box, there is a wide variance in how "fair" those surrogates appear. A bad actor can deliberately select a surrogate that looks ethically clean on the surface. If this surrogate suffers from low fidelity on a minority subgroup, it will effectively smooth over and hide the model's reliance on sensitive attributes for that specific population (Dai et al.).

[Biased Black-Box Model] ──(High Fidelity on Majority)──> [Surrogate Explanation] (Looks Fair)
           │
           └──(Low Fidelity on Minority)───> [Obscures Discriminatory Logic]

Fortunately, the mathematical relationship between fidelity and bias also provides a tool for detection. Shamsabadi et al. prove that fairwashing inherently creates an unavoidable, detectable "divergence" in how well an explanation mimics a model across different demographic groups. Because a fairwashed explanation must lie about the model's true logic for the penalized group, its approximation accuracy will inevitably drop for that specific subpopulation.

Beyond Aggregate Metrics

The core takeaway from this domain is that evaluating explainable AI requires moving away from aggregate validation. To ensure that the right to an explanation is upheld equitably (Kern et al.), auditors must implement subgroup explanation fidelity measurements (Mhasawade et al.). Standardizing these evaluations through open-source ecosystems like OpenXAI (Agarwal et al.) allows practitioners to verify that transparency tools are equally truthful for all individuals, preventing biased models from hiding behind a facade of global average faithfulness.

Key Foundational Work

The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations (2022) — contributed 6 concept elements to this direction.
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations (2022) — contributed 5 concept elements to this direction.
Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022) (2022) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_fidelity_disparity_e2cd86fa,
  title = {explanation fidelity disparity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-fidelity-disparity/rdr_e2cd86faf3504c40a4ce0e17cd96c2f3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_explanation_fidelity_disparity_e2cd86fa_v2,
  title = {explanation fidelity disparity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-fidelity-disparity/rdr_e2cd86faf3504c40a4ce0e17cd96c2f3/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake