Measuring Interpretability Faithfulness with Synthetic Ground Truth

This direction is merged into another direction: explanation quality metrics. Its last version stays readable and citable here.

Version 1 of 154 papers59 conceptsfrozen 2026-10-04

Deep neural networks are often described as "black boxes," and the field of Explainable AI (XAI) has produced dozens of methods—like saliency maps, SHAP, and Integrated Gradients—to peek inside. However, evaluating these explanations is notoriously difficult. If a heatmap highlights a specific region of an image, is it because the model actually used that feature, or because the explanation method is simply "hallucinating" a plausible reason?

To move beyond subjective "looks good to me" evaluations, researchers are increasingly using synthetic ground truth. By programmatically designing datasets where the relationship between features and labels is mathematically defined, we can finally measure whether an explanation is truly "faithful" to the model's internal logic.

The Problem of Subjective Evaluation

In natural datasets, we rarely know the "true" reason a model makes a prediction. A model might classify a "husky" correctly because of its fur, or it might be using a "shortcut" like the snow in the background. If an explanation method highlights the fur, we might praise it, even if the model was actually looking at the snow.

To solve this, Yilun Zhou and colleagues proposed a "unit test" framework in Do Feature Attribution Methods Correctly Attribute Features?. They modified datasets by randomizing original labels and injecting artificial, perfectly correlated artifacts. Because the original features become non-predictive, any high-performing model must rely on these injected artifacts. This creates a definitive ground truth: if the explanation method fails to highlight the artifact, it is objectively unfaithful.

Mechanisms for Creating Ground Truth

Researchers have developed several distinct strategies to build these controlled environments:

1. Feature Injection and Shortcuts

In NLP, Polina Zablotskaia and her team introduced a protocol in "Will You Find These Shortcuts?" that injects synthetic lexical shortcuts into text classification tasks. By verifying the model's reliance on these known tokens, they can measure the precision and recall of various salience methods. Similarly, in medical imaging, Marta Oliveira and Fabian Eitel used synthetic lesions—programmatically generated shapes overlaid on real brain MRIs—to provide pixel-perfect masks for evaluating attribution accuracy (Benchmarking the influence of pre-training on explanation performance in MR image classification).

2. Symbolic and Mathematical Labeling

For tabular data, Brian Barr and colleagues developed a framework that uses symbolic mathematical expressions to define target labels (Towards Ground Truth Explainability on Tabular Data). By applying a function like $y = x_1 \cdot x_2$ to a subset of features, they create a deterministic relationship where the "importance" of each variable is mathematically provable. This allows for the objective evaluation of methods like SHAP against a known gold standard.

3. Adversarial Backdoors as Ground Truth

An ingenious approach involves repurposing adversarial attacks. Jimin Sun and May D. Wang demonstrated that "poisoning" a model with a hidden backdoor trigger creates a known ground truth for XAI (Improving explainable AI with patch perturbation-based evaluation pipeline). If a model is trained to classify any image with a specific "trigger" patch as a certain class, a faithful explanation must identify that patch. Mengxi Ya further refined this with Generalization-Limited Backdoor Watermarks (GLBW) to force the model's focus strictly onto the trigger (Towards Faithful XAI Evaluation via Generalization-Limited Backdoor Watermark).

4. Structured Logic and Solved Games

Sometimes the best ground truth comes from domains with rigid rules. Adam Karvonen and Samuel D. Marks used board game models to evaluate dictionary learning and Sparse Autoencoders (Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models). Because the state of a board game is perfectly defined by its rules, researchers can quantitatively measure how accurately a model's internal "features" capture the actual game state. Similarly, Stephan Waeldchen used Connect Four to train Reinforcement Learning agents, providing a mathematically defined "characteristic function" for feature importance (Training Characteristic Functions with Reinforcement Learning).

Why This Works: Verification by Design

The core intuition behind this research direction is that interpretability requires a reference point. Without a ground truth, we are merely comparing one black box (the model) to another (the explainer).

By controlling the data generation process, researchers can:

  • Isolate Causal Features: Ensure that only specific features are predictive, eliminating the ambiguity of correlated noise (Benedict Clark and Rick Wilming in XAI-TRIS).
  • Quantify Localization: Use metrics like Intersection over Union (IoU) to measure how well a heatmap overlaps with the actual feature coordinates (Jose Oramas in Visual Explanation by Interpretation).
  • Test Model Logic: Yang Zhang and colleagues even suggested that controlling data isn't enough; we must also control model weights to ensure the explanation matches the actual decision process (AttributionLab).

Conclusion

The shift toward synthetic benchmarking represents a maturation of the interpretability field. As platforms like OpenXAI (neurips 2022) and EXACT (arxiv 2024) standardize these synthetic evaluations, we move away from qualitative "sanity checks" toward a rigorous, reproducible science of model understanding.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_measuring_interpretability_faithfulness__b7aa6950,
  title = {Measuring Interpretability Faithfulness with Synthetic Ground Truth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/measuring-interpretability-faithfulness-with-synthetic-ground-truth/rdr_b7aa6950f1b64da190c955a7b6893d54},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_measuring_interpretability_faithfulness__b7aa6950_v1,
  title = {Measuring Interpretability Faithfulness with Synthetic Ground Truth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/measuring-interpretability-faithfulness-with-synthetic-ground-truth/rdr_b7aa6950f1b64da190c955a7b6893d54/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}