As language models (LMs) move from being curiosities to critical tools in fields like healthcare, law, and fact-checking, a pressing question remains: Why did the model say that?
In her Ph.D. thesis, Jingyi Sun explores the messy reality of AI explainability. While we have various ways to make models "explain" themselves—ranging from highlighting important words to generating full-paragraph justifications—these explanations are often "unfaithful." That is, a model might give you a perfectly readable reason that has absolutely nothing to do with the actual math it used to reach its conclusion.
TL;DR
This thesis provides a comprehensive toolkit for evaluating and improving how language models explain their reasoning. By bridging the gap between simple "highlight" explanations and complex natural language rationales, Sun demonstrates how we can make AI more transparent, faithful to its own internal logic, and ultimately more useful for human decision-makers.
What Problem Does This Paper Address?
The core challenge in Explainable AI (XAI) is the tension between faithfulness (how accurately an explanation reflects the model's internal process) and plausibility (how convincing and readable it is to a human).
Current models are great at being plausible but often fail at being faithful. They might "hallucinate" a logical-sounding rationale for a biased or incorrect prediction. Furthermore, the field lacks unified ways to measure these qualities across different types of explanations, such as:
- Highlight Explanations: Identifying which specific words in the input were most important.
- Natural Language Explanations (NLEs): Free-text sentences justifying a verdict.
- Chain-of-Thought (CoT): Step-by-step reasoning traces produced before the final answer.
Sun’s work addresses the lack of rigorous evaluation frameworks and the difficulty of generating explanations that are both human-friendly and technically accurate.
Methods
Sun introduces several novel frameworks and techniques:
- Unified Diagnostic Evaluation: A framework to compare different types of "highlights" (individual tokens vs. interactions between spans of text) across four dimensions: faithfulness, human agreement, simulatability, and complexity.
- G-TEx (Graph-Guided Textual Explanation): A method that extracts faithful "highlights" from a model and encodes them into a graph. This graph then guides the model to generate a natural language explanation that is anchored to the actual features the model used.
- CLUE (Conflict- & Agreement-aware Uncertainty Explanations): A "white-box" pipeline for fact-checking. It identifies where different pieces of evidence conflict or agree and steers the model to explain its uncertainty based on those specific spans.
- FaithMATE: A meta-evaluation framework using Direct Preference Optimization (DPO) to test whether "faithfulness" is a single trait. It asks: if we train a model to be more faithful according to one metric, does it actually become more faithful according to others?
Datasets
The research utilizes a wide array of benchmarks to ensure robustness:
- Fact-Checking: FEVER, HealthVer, and DRUID (real-world claims with retrieved evidence).
- Reasoning & Inference: SNLI, e-SNLI (NLI with human explanations), and ECQA (commonsense QA).
- Context Utilization: World Capital, CounterFact, Fakepedia, and ConflictQA (used to test if models rely on provided text vs. their own internal "memory").
Key Insights and Ideas
- The "Aha" Moment on Spans: The thesis finds that span-level interactions (how two phrases relate) are far more effective for human understanding and "simulatability" than just highlighting individual words.
- Faithfulness is Not Monolithic: Through the FaithMATE framework, Sun discovered that different faithfulness metrics often capture entirely different behaviors. Improving a model’s score on one metric rarely guarantees it will improve on another, suggesting we need a "family" of metrics for a true evaluation.
- Evidence is King: In a user study on fact-checking, Sun found that humans rely on raw evidence more than the AI’s explanation. However, explanations are highly valued as "error detectors"—when the AI's explanation contradicts the evidence, users immediately become more skeptical.
- Positional Bias: Models have a "hidden" bias where they favor information at the very beginning or very end of a prompt. Current explanation methods often fail to reveal this mechanical quirk.
Key Results
- G-TEx improved the faithfulness of natural language explanations by up to 12.18% over standard fine-tuning.
- CLUE generated uncertainty explanations that human raters judged as more helpful and less redundant than standard prompting.
- MechLight (a mechanistic interpretability approach) outperformed traditional gradient-based methods in identifying which specific documents a model actually used to answer a question.
Limitations and Future Work
Sun identifies several areas for the next generation of XAI researchers:
- Scaling Up: Most experiments were conducted on medium-sized models (T5, BART, Llama-3-8B). Testing these frameworks on "frontier" models (like GPT-4 or Claude 3) is a necessary next step.
- Beyond Text: The current work is text-based. Future research should look at multimodal explanations (e.g., explaining a medical diagnosis based on both a scan and a lab report).
- The "Thinking Route" Problem: It remains unclear if models have a single, deterministic "thinking route" that an explanation can uncover, or if reasoning is distributed across billions of parameters in a way that can never be fully verbalized.
By moving from "black box" predictions to "glass box" reasoning, this thesis brings us one step closer to AI that we can not only use but truly understand.
Summary updated