Abstract
As Large Language Models (LLMs) transition from simple classifiers to complex generative systems, the methodologies used to explain their behavior have reached a critical crossroads. This paper provides a comprehensive roadmap for "Explanation" in the LLM era, charting the shift from traditional feature-attribution methods to modern generative approaches like Chain-of-Thought (CoT). We categorize the opportunities LLMs provide for self-rationalization while highlighting the inherent tension between making an explanation "plausible" to humans versus "faithful" to the model's internal logic.
Introduction
For years, eXplainable AI (XAI) focused on "post-hoc" methods—techniques designed to peek under the hood of a finished prediction to identify which input features were most influential. However, the rise of LLMs has introduced three fundamental challenges that render many traditional methods obsolete. First is Scale: importance-score methods are often computationally prohibitive for models with hundreds of billions of parameters. Second is Opacity: many state-of-the-art models are closed-source and accessible only via API, making gradient-based or mechanistic explanations impossible for the broader research community. Finally, there is the Plausibility vs. Faithfulness Gap: LLMs are so adept at generating human-like text that they can produce convincing justifications that appear correct to a human (plausible) but do not actually reflect the model's decision-making process (unfaithful).
A Taxonomy of LLM Explanation
The current landscape of explanation methods can be organized into four primary pillars:
- Free-text and Chain-of-Thought (CoT): These methods leverage the model’s own generative power to produce natural language justifications. This includes "self-rationalizing" models that output a reasoning path and a prediction simultaneously, potentially making the reasoning more integral to the final output.
- Structured Explanations: To mitigate the ambiguity of raw text, researchers are moving toward more rigorous formats such as Trees-of-Thought, Graph-of-thoughts, or symbolic rules. These structures help ensure logical consistency and allow for more formal verification of the reasoning process.
- Importance Scores: This traditional pillar remains relevant through token-wise attribution (identifying which words mattered) and instance-wise attribution (identifying which training examples influenced a specific output). However, techniques like influence functions are becoming increasingly expensive as training sets scale.
- Mechanistic and Causal Interpretability: Often described as the "neuroscience" of AI, this involves identifying specific "neural circuits" within the transformer architecture. It also encompasses Model Editing (e.g., ROME or MEND), where researchers modify specific weights to observe how a model's internal "knowledge" of a fact changes.
Evaluating Explanations: The Desiderata
A central theme in modern XAI is the dual role of LLMs: they are both the subject of explanation (we want to know why they said X) and the tool for explanation (we use them to explain other complex systems). To evaluate the quality of these explanations, the authors propose four essential properties:
- Faithfulness: The explanation must accurately reflect the model's internal reasoning.
- Plausibility: The explanation must be convincing and understandable to a human user.
- Usefulness: The explanation should assist the user in achieving a specific task or goal.
- Minimality: The explanation should include only the necessary factors, avoiding extraneous information.
Benchmarks such as ERASER, WorldTree, and StrategyQA have become vital for measuring these properties, though the field currently faces an "evaluation crisis." Relying on LLMs to evaluate other LLMs (LLM-as-a-judge) introduces new biases that the community has yet to fully resolve.
Discussion and Future Directions
While self-rationalization and CoT offer promising paths forward, they are not silver bullets. Research indicates that models can still provide unfaithful reasoning chains that do not match their internal computations. Furthermore, the shift toward proprietary, closed-source models limits the ability to perform deep mechanistic interpretability, pushing the field toward "black-box" behavioral analysis.
Future work must focus on "context faithfulness"—ensuring models rely on provided evidence rather than internal training biases—and developing more robust prompt-engineering methods to elicit reliable reasoning. As LLMs become more integrated into high-stakes decision-making, bridging the gap between what a model says and what a model "thinks" remains the most pressing challenge in AI safety and transparency.
Summary updated