Introduction
In the rapidly evolving field of Natural Language Processing (NLP), the quest for explainability has largely been viewed as a technical challenge: how can we peer inside the "black box" of large language models (LMs)? However, this position paper argues that technical transparency is only half the battle. The authors introduce the concept of the "rationalization trap," a phenomenon where humans project their own linguistic intuitions onto models, leading to a false sense of understanding.
True explainability in NLP is as much a Human-Computer Interaction (HCI) problem as it is a computational one. The fundamental issue lies in a misalignment between how machines and humans process language. While LMs learn statistical patterns and logical decision boundaries (form), humans perceive language through a lens of semantics, experience, and intent (meaning). Because language is "deceivably intelligible," we are prone to assuming we understand a model's reasoning simply because the output is in a format we use every day.
A Conceptual Framework for NLP Explainability
To address this misalignment, the authors propose a conceptual framework that situates the explanation process between the Language Model and the User's Mental Model. This framework acknowledges that a user's background—whether rooted in computational linguistics or theoretical grammar—significantly influences how they interpret an explanation.
The framework identifies three pillars for effective explanation:
- High Fidelity: The explanation must accurately reflect the model's internal mechanics.
- Completeness: Partial information can be more dangerous than no information, as it invites users to "fill in the blanks" with their own biases.
- User Fit: Explanations must be tailored to calibrate the user's specific mental model, moving them toward a machine-centric logic when necessary.
Navigating the Decision Tree of Understanding
The core of the paper's contribution is a decision tree model that maps the potential outcomes of an XAI interaction. This model highlights the precarious paths that lead to either "Truthful Understanding" or the "Rationalization Trap."
The Rationalization Trap and Its Consequences
The "rationalization trap" occurs when an explanation is low-fidelity or incomplete, yet the user finds a way to make it "make sense" using their own world knowledge. This is particularly dangerous in high-stakes environments because a plausible-sounding explanation can increase a user's trust in a model even when that model is behaving erratically or incorrectly.
The authors note that popular methods like saliency maps (feature attribution) often fall into this trap. While they are used in nearly half of NLP explainability research, empirical studies suggest they rarely help users accurately predict or simulate model behavior. Instead, they often serve as visual "justifications" that users interpret through their own linguistic biases.
Moving Toward Truthful Understanding
The paper concludes by defining a roadmap for future XAI research. The goal should not just be "more" explanation, but "better" calibration.
- Path to Success: Achieving "Truthful Understanding" requires high-fidelity, complete explanations that successfully align the user's mental model with the model's actual behavior.
- Path to Failure: When explanations fail to match or calibrate the user, the result is either "Untruthful Rationalization" (the user is confidently wrong) or "No Truthful Understanding" (the user remains skeptical and un-informed).
Future Directions
The authors call for empirical user studies to quantify the rationalization trap and the development of "Adaptive XAI." Such systems would ideally detect when a user is misinterpreting an explanation and provide natural language feedback to correct their mental model. By treating explainability as a dynamic, human-centered process, the field can move beyond the illusion of understanding toward genuine, truthful insights into machine behavior.
Summary updated