Abstract
This paper formalizes Saliency Map Verbalization (SMV)—the task of translating raw feature importance scores (saliency maps) into readable natural language. By comparing a fast, model-free template approach against a GPT-3.5 instruction-based method, the researchers found that while LLMs produce significantly more helpful and understandable explanations, they often struggle with numeric faithfulness and "hallucinate" reasoning that isn't present in the underlying model.
Introduction
In the field of Explainable AI (XAI), saliency maps are the standard tool for identifying which input features (e.g., words) most influence a model's prediction. Typically visualized as heatmaps, these explanations often suffer from a "usability gap": they can be cognitively taxing for non-experts to interpret and are prone to visual confirmation biases. The authors address this by exploring verbalization: the conversion of raw importance scores into natural language sentences. This task involves determining both what to say (content selection) and how to say it (surface realization) while maintaining faithfulness to the original model.
Methodology
The study proposes and evaluates two distinct approaches to SMV:
- Model-free SMV (Template-based): This method utilizes search algorithms to identify the most salient snippets of text. Convolution Search looks for clusters of important tokens, while Span Search identifies contiguous blocks. These snippets are ranked using metrics such as Weighted Average or Quantile and then inserted into human-readable templates (e.g., "The phrase [X] was most important for the [Y] prediction").
- Instruction-based SMV (GPT-3.5): This zero-shot approach leverages a Large Language Model (LLM). The prompt includes the input text alongside its saliency scores (e.g.,
movie (0.15)), and the LLM is tasked with explaining the analyzer's prediction in simple language.
Evaluation and Results
The researchers evaluated these methods using the IMDb (sentiment analysis) and AG News (topic classification) datasets. A BERT model served as the subject, with saliency maps generated via Integrated Gradients. A human study with 10 participants measured Simulatability, Helpfulness, and Ease of Understanding.
Key findings include:
- Ease of Understanding: SMV-GPT was the clear favorite, scoring 5.88/7 on IMDb, significantly higher than the 4.35/7 achieved by traditional heatmaps.
- Simulation Accuracy: On the IMDb dataset, humans using GPT-generated explanations correctly predicted the model's behavior 94.06% of the time.
- Faithfulness vs. Plausibility: While GPT-3.5 produces highly readable explanations, it often "hallucinates" reasoning or incorporates external world knowledge not present in the model's actual logic. In contrast, the template-based method (SMV-Templ) proved significantly more faithful to the underlying numeric saliency scores.
Discussion and Limitations
The research highlights a critical tension in XAI: explanations that are easy for humans to understand (plausible) are not always faithful to the model's internal mechanics. Furthermore, efficiency is a major differentiator; the template-based method can process 25,000 instances in under two minutes on a standard CPU, whereas LLM-based methods are computationally expensive and have a larger carbon footprint.
Limitations of the current work include the potential for LLMs to flip the meaning of text (e.g., changing "not reliable" to "unreliable"), which can mislead users. Future research is needed to extend SMV to more complex tasks like Question Answering and to explore the use of smaller, specialized models to reduce dependency on closed-source black-box systems.
Summary updated