Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language

Rita Sevastjanova, Mennatallah El-AssadyarXiv 2022 · DOI · arXiv

Introduction

In the rapidly evolving field of Natural Language Processing (NLP), the quest for explainability has largely been viewed as a technical challenge: how can we peer inside the "black box" of large language models (LMs)? However, this position paper argues that technical transparency is only half the battle. The authors introduce the concept of the "rationalization trap," a phenomenon where humans project their own linguistic intuitions onto models, leading to a false sense of understanding.

True explainability in NLP is as much a Human-Computer Interaction (HCI) problem as it is a computational one. The fundamental issue lies in a misalignment between how machines and humans process language. While LMs learn statistical patterns and logical decision boundaries (form), humans perceive language through a lens of semantics, experience, and intent (meaning). Because language is "deceivably intelligible," we are prone to assuming we understand a model's reasoning simply because the output is in a format we use every day.

A Conceptual Framework for NLP Explainability

To address this misalignment, the authors propose a conceptual framework that situates the explanation process between the Language Model and the User's Mental Model. This framework acknowledges that a user's background—whether rooted in computational linguistics or theoretical grammar—significantly influences how they interpret an explanation.

The conceptual framework illustrating the interplay between Language Models, Explanations, and the diverse Mental Models of users.

The framework identifies three pillars for effective explanation:

  1. High Fidelity: The explanation must accurately reflect the model's internal mechanics.
  2. Completeness: Partial information can be more dangerous than no information, as it invites users to "fill in the blanks" with their own biases.
  3. User Fit: Explanations must be tailored to calibrate the user's specific mental model, moving them toward a machine-centric logic when necessary.

The core of the paper's contribution is a decision tree model that maps the potential outcomes of an XAI interaction. This model highlights the precarious paths that lead to either "Truthful Understanding" or the "Rationalization Trap."

A decision tree mapping the cognitive paths from an explanation to either truthful understanding or untruthful rationalization.

The Rationalization Trap and Its Consequences

The "rationalization trap" occurs when an explanation is low-fidelity or incomplete, yet the user finds a way to make it "make sense" using their own world knowledge. This is particularly dangerous in high-stakes environments because a plausible-sounding explanation can increase a user's trust in a model even when that model is behaving erratically or incorrectly.

The authors note that popular methods like saliency maps (feature attribution) often fall into this trap. While they are used in nearly half of NLP explainability research, empirical studies suggest they rarely help users accurately predict or simulate model behavior. Instead, they often serve as visual "justifications" that users interpret through their own linguistic biases.

Moving Toward Truthful Understanding

The paper concludes by defining a roadmap for future XAI research. The goal should not just be "more" explanation, but "better" calibration.

  • Path to Success: Achieving "Truthful Understanding" requires high-fidelity, complete explanations that successfully align the user's mental model with the model's actual behavior.
  • Path to Failure: When explanations fail to match or calibrate the user, the result is either "Untruthful Rationalization" (the user is confidently wrong) or "No Truthful Understanding" (the user remains skeptical and un-informed).

Future Directions

The authors call for empirical user studies to quantify the rationalization trap and the development of "Adaptive XAI." Such systems would ideally detect when a user is misinterpreting an explanation and provide natural language feedback to correct their mental model. By treating explainability as a dynamic, human-centered process, the field can move beyond the illusion of understanding toward genuine, truthful insights into machine behavior.

Summary updated

Abstract

Language models learn and represent language differently than humans; they learn the form and not the meaning. Thus, to assess the success of language model explainability, we need to consider the impact of its divergence from a user's mental model of language. In this position paper, we argue that in order to avoid harmful rationalization and achieve truthful understanding of language models, explanation processes must satisfy three main conditions: (1) explanations have to truthfully represent the model behavior, i.e., have a high fidelity; (2) explanations must be complete, as missing information distorts the truth; and (3) explanations have to take the user's mental model into account, progressively verifying a person's knowledge and adapting their understanding. We introduce a decision tree model to showcase potential reasons why current explanations fail to reach their objectives. We further emphasize the need for human-centered design to explain the model from multiple perspectives, progressively adapting explanations to changing user expectations.
BibTeX
@misc{sevastjanova2022beware,
  author = {Rita Sevastjanova and Mennatallah El-Assady},
  title = {{Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language}},
  year = {2022},
  eprint = {2207.06897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.06897},
}
Figure