Abstract
While interpreting classification models has become a standard practice in Explainable AI (XAI), explaining sequence generation models—particularly in the context of dialogue—remains an under-explored and difficult problem. Dialogue response generation is uniquely challenging due to its open-ended nature and the lack of direct word-to-word alignment between inputs and outputs. We propose Local Explanation of Response Generation (LERG), a model-agnostic method that treats explanations as the mutual interaction between segments of input and output sentences. By viewing sequence prediction as an uncertainty estimation of a human response, LERG creates explanations by perturbing the input and calculating the resulting change in certainty. Our method satisfies key theoretical properties: unbiased approximation, consistency, and cause identification. Empirically, LERG improves upon existing methods by 4.4–12.8% across various metrics and demonstrates a superior ability to extract both explicit and implicit relations.
The Challenge of Dialogue Explanation
Explaining why a chatbot generates a specific response is significantly harder than explaining a classifier's decision. In classification, the output space is discrete and limited. In dialogue, the output space is exponential, and the relationship between input and output is often implicit. For example, a user mentioning a "hot potato" might trigger a response about "difficult situations" or "gasoline prices" depending on context; these conceptual links are not easily captured by traditional alignment-based XAI tools.
As shown in the figure above, traditional classifier explanations (b) focus on mapping input features to a single label. In contrast, LERG (c) identifies how specific segments of the input dialogue influence specific segments of the generated output, effectively uncovering the "intent" behind the response.
The LERG Framework
LERG is a model-agnostic framework that evaluates the "certainty" a model has regarding a ground-truth human response. By masking segments of the input and measuring how the model's certainty fluctuates, LERG assigns importance scores to input-output pairs. The paper introduces two implementations:
- LERG_L: A learning-based approach using a linear proxy to approximate model behavior locally.
- LERG_S: A game-theoretic approach based on Shapley values that calculates the unique contribution of each input segment.
Theoretical analysis confirms that LERG adheres to three essential properties for text generation explanations: unbiased approximation (the sum of segment contributions equals the total model gain), intra-response consistency, and causal cause identification.
Empirical Results
The researchers evaluated LERG on the DailyDialog, PersonaChat, and XSum datasets. The method was compared against standard baselines including Attention, Gradient-based methods, LIME, and vanilla Shapley values.
Key findings include:
- Quantitative Gains: LERG outperformed traditional methods by 4.4% to 12.8% on automatic evaluation metrics.
- Human Evaluation: In a "sufficiency" test, humans were shown only the top 20% of input tokens identified by LERG. They were able to correctly predict the intended response 56% of the time, significantly higher than the ~35% achieved by Attention-based methods.
- Implicit Reasoning: LERG successfully identified non-obvious triggers, such as recognizing that a question mark in the input was the primary cause for a "Yes" or "No" in the output.
Conclusion
LERG provides a robust, theoretically grounded framework for explaining sequence generation. By focusing on the interaction between input and output segments, it moves beyond simple word-matching to capture the conceptual reasoning of dialogue models. Future work will look into hierarchical explanations to better cluster multi-word phrases and using explainability as a core metric for evaluating model rationality.
Summary updated