Bridging the Interpretability Gap with Natural Language Narratives

This direction is split into other directions: explanation utility gap. Its last version stays readable and citable here.

Version 1 of 1231 papers252 conceptsfrozen 2026-10-04

The field of Explainable AI (XAI) has long focused on "opening the black box," but a recurring realization across recent research is that simply showing the gears inside does not help the person driving the car. While techniques like SHAP, LIME, and saliency maps provide mathematically rigorous feature importance scores, they often fail to provide actionable insights for the non-experts who actually use these systems.

The emerging research direction focuses on Verbalization and Rationalization: the process of translating raw technical signals—like neural network weights or gradient vectors—into human-centric, natural language narratives grounded in domain knowledge.

The Communication Gap in Technical XAI

Traditional interpretability tools are designed by and for data scientists. As Azzurra Ragone and Fedelucio Narducci point out in A General Model for Fair and Explainable Recommendation in the Loan Domain, there is a significant gap between providing a mathematical feature weight and delivering an explanation a consumer can actually understand.

Consider a doctor using an AI to detect pneumonia. A saliency map might highlight a cluster of pixels in a chest X-ray. However, as Patrick Shafto and colleagues argue in their work on Bayesian Teaching, these sub-symbolic representations fail to bridge the gap to the high-level symbolic concepts—like "opacity" or "pleural effusion"—that medical professionals use to communicate.

From Feature Weights to High-Level Concepts

To solve this, researchers are moving away from explaining features (raw data inputs) and toward explaining concepts.

Catarina G Belém and her team introduced JOEL (Jointly learned cOncept-based ExpLanations) in Teaching the Machine to Explain Itself using Domain Knowledge. Instead of just predicting a result, the model simultaneously predicts high-level domain concepts. In fraud detection, for instance, the model doesn't just flag a transaction; it explains it through concepts like "unusual location" or "velocity of spending," which are immediately intelligible to human investigators.

Similarly, in the realm of autonomous vehicles, Daniel Omeiza and his team developed a system that mimics "commentary driving". Rather than showing a heatmap of sensor activations, the AI verbalizes its observations and intentions ("I am slowing down because the pedestrian is near the curb"), mimicking the narrative-style reasoning humans use to build trust.

Verbalization via Large Language Models

The rise of Large Language Models (LLMs) has provided the "missing link" for translating math into prose. Researchers are now using LLMs as a translation layer that sits on top of traditional XAI outputs.

  1. Saliency Map Verbalization (SMV): Leonhard Hennig and colleagues formalized the task of translating raw importance scores into natural language. While they found that LLMs can make explanations more "helpful," they also warned of a trade-off: LLMs sometimes prioritize human-friendly plausibility over technical faithfulness.
  2. Theory-Driven Feedback: In the educational domain, the iLLuMinaTE framework, developed by Vinitra Swamy and Tanja Käser, uses LLMs to turn technical student performance data into actionable feedback. By grounding the LLM in social science theories, the system ensures the explanation isn't just a summary of data, but a pedagogical tool.
  3. Traceable QA: Recent work has integrated Retrieval-Augmented Generation (RAG) to create "traceable" dialogues. By grounding LLMs in a structured repository of feature importance and counterfactuals, users can ask "Why?" in a natural conversation and receive answers that are both linguistically fluid and mathematically grounded.

Explanations as a Social Process

A critical insight driving this direction is that human explanation is not a data dump; it is a social interaction. Brent Mittelstadt argues in Explaining Explanations in AI that for AI to be truly accountable, it must shift from "scientific modeling" to "everyday explanations."

According to this view, a good explanation should be:

  • Contrastive: Explaining why "X" happened instead of "Y".
  • Selective: Focusing on the most relevant causes rather than a complete list of every variable.
  • Social: Packaged in a way that fits the user's context and technical literacy.

This shift is evident in tools like CEB (Counterfactual Examples for Bias) by Chelsea M. Myers, which allows non-experts to interactively "flip" features to see how a model's decision changes. By moving from static charts to interactive, contrastive "what-if" scenarios, the explanation becomes a process of discovery rather than a passive report.

Conclusion

The goal of this research is to move beyond "transparency" (seeing the code) toward "comprehensibility" (understanding the logic). By leveraging LLMs, domain-specific concepts, and social science theories, the field is building a new generation of AI that doesn't just show its work in numbers, but justifies its decisions in the language of the humans it serves.

Papers in this version

Showing 60 of 231 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_interpretability_gap_with_n_e365854e,
  title = {Bridging the Interpretability Gap with Natural Language Narratives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-interpretability-gap-with-natural-language-narratives/rdr_e365854e8c7e45bf9cf9fe5fa3046f40},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_interpretability_gap_with_n_e365854e_v1,
  title = {Bridging the Interpretability Gap with Natural Language Narratives},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-interpretability-gap-with-natural-language-narratives/rdr_e365854e8c7e45bf9cf9fe5fa3046f40/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}