This NeurIPS 2021 paper, "Local Explanation of Dialogue Response Generation," tackles the "black box" problem in conversational AI. While we’ve become quite good at explaining why a model classifies an image as a "cat," explaining why a chatbot chooses a specific, open-ended sentence is a much steeper mountain to climb.
TL;DR
The authors introduce LERG (Local Explanation of Response Generation), a model-agnostic framework that identifies which parts of a prompt most heavily influence specific parts of a dialogue response. By treating sequence generation as an uncertainty estimation problem, LERG successfully uncovers both explicit word alignments and implicit relationships (like sociability and implication) that traditional methods miss.
What Problem Does This Paper Address?
Most "Local Explanation" methods (like LIME or Shapley values) were designed for classification tasks with a small, fixed number of labels. Dialogue generation is fundamentally different: the output space is effectively infinite, and responses are often "alignment-free."
For example, if a user asks, "How are you today?" and the model responds, "Good. Thank you for asking," there isn't a direct word-to-word mapping between the input and output. Traditional tools struggle here because they can't easily handle sequential outputs or the subtle, implicit logic required for human-like conversation.
The gap the authors address is the lack of a systematic way to explain why a generative model thinks a specific response is certain, moving beyond the simple input-to-label mapping of traditional classifiers.
Methods
The core innovation is LERG, which views sequence prediction as the "certainty estimation" of a human response. Instead of explaining the model's own (potentially flawed) sampled output, LERG looks at the ground-truth human response and asks: “How does my certainty about this specific response change if I perturb the input?”
The authors propose two variations:
- LERG_L: A learning-based approach similar to LIME that fits a linear proxy to simulate how input segments influence output segments.
- LERG_S: A variation inspired by Shapley values that calculates the marginal contribution of each input segment.
To ensure these explanations are actually useful, the authors define three theoretical requirements for text generation explanations: unbiased approximation, intra-response consistency, and causal cause identification.
Datasets
The primary benchmark used is DailyDialog, a high-quality dataset of human-to-human conversations covering daily topics. It was chosen because its responses are informative and less "robotic" than other datasets. The authors fine-tuned GPT and DialoGPT models on this data to serve as the "black boxes" being explained.
Key Insights and Ideas
- Beyond Word Alignment: The most striking insight is that LERG can find "implication" and "sociability" links. For instance, it can show that the phrase "hot potato" in a prompt is the primary driver for the word "gasoline" in a response about rising prices—a link no simple keyword-matching algorithm would find.
- Explaining Certainty, Not Just Output: By focusing on the probability of the human response, the method avoids the "compounding error" problem where a model's own early mistakes in a sentence make the rest of the sentence impossible to explain.
- Necessity vs. Sufficiency: The authors formalize how to evaluate an explanation. Necessity asks: "If I remove the 'important' words, does the model get much more confused?" Sufficiency asks: "If I only give the model the 'important' words, can it still predict the response?"
Key Results
LERG consistently outperformed baselines (LIME, Shapley, Attention, and Gradient-based methods) on automatic metrics by 4.4% to 12.8%.
In a user study for human evaluation, participants were shown only the top 20% of input tokens identified by LERG as "salient." Using just those fragments, humans could correctly identify the intended response from a set of candidates with 56% accuracy (compared to ~25% for random selection). Furthermore, LERG_S proved to be particularly "faithful," meaning the explanation truly reflects the underlying model's logic rather than just providing a "human-readable" guess.
Limitations and Future Work
The authors acknowledge that LERG isn't perfect. Their error analysis shows that the method sometimes identifies words that are related to the correct concept but aren't the best explanation (e.g., linking "carry" to "bags" and "suitcases" but failing to cluster them into a single "luggage" concept).
Looking ahead, the authors suggest that these explanations could eventually be used as a new type of evaluation metric for dialogue models. Instead of just checking if a chatbot's answer is "correct," we could check if it arrived at that answer for the "right" reasons. They also point toward "hierarchical explanations" that could group individual words into higher-level concepts.
Summary updated