A Necessary Step toward Faithfulness: Measuring and Improving Consistency in Free-Text Explanations

Lingjun Zhao, Hal Daumé2025 · DOI

Abstract

Large Language Models (LLMs) excel at generating plausible explanations that sound convincing to humans but may not reflect the model's actual reasoning process. This paper addresses the gap between plausibility and faithfulness by focusing on logical consistency. The authors introduce Prediction-EXplanation (PEX) consistency, a metric based on the weight of evidence framework. By applying Direct Preference Optimization (DPO) with PEX as a reward signal, they significantly reduce logical inconsistencies. Experimental results on opinion spam detection datasets show that improving consistency leads to a measurable increase in explanation faithfulness, with simulatability gains of up to 9.7%.

Introduction: The Plausibility-Faithfulness Gap

In the field of Explainable AI (XAI), a critical distinction exists between plausibility—how convincing an explanation is to a human—and faithfulness—how accurately it reflects a model's internal reasoning. While modern Large Language Models (LLMs) are highly plausible, they often lack faithfulness. A primary indicator of this unfaithfulness is logical inconsistency, where a model provides the same justification for two contradictory predictions.

The Problem of Inconsistent Explanations

The authors identify a "double-talk" phenomenon where LLMs use identical evidence to support both a conclusion and its negation. For instance, in a hotel review classification task, a model might cite "overwhelmingly positive tone" as evidence for a truthful review, while simultaneously citing "exaggeratedly positive language" as evidence for a deceptive one. If an explanation can support both a prediction $y$ and its negation $\neg y$, it fails as a faithful deductive account.

Example of an inconsistent explanation where similar reasoning is used to justify contradictory predictions.

Measuring and Improving Consistency via PEX and DPO

To address this, the researchers propose Prediction-EXplanation (PEX) consistency. PEX measures the log-likelihood ratio of an explanation being generated for a prediction $y$ versus its negation $\neg y$. A high PEX score indicates the explanation strongly favors one prediction over the other, serving as a robust weight of evidence.

The improvement phase utilizes Direct Preference Optimization (DPO). By sampling multiple explanations and ranking them by PEX scores, the model is fine-tuned to prefer consistent justifications. This process effectively "teaches" the model to generate evidence that uniquely supports its chosen prediction rather than generic, plausible-sounding text.

The PEX-DPO workflow: sampling explanations, ranking by consistency, and fine-tuning via DPO.

Evaluating Faithfulness through Simulatability

To verify if consistency implies faithfulness, the study uses a student-teacher framework. A "teacher" model generates predictions and explanations. A "student" model then attempts to predict the teacher's original label using only the provided explanation. Higher student accuracy (simulatability) indicates that the explanation contains the necessary logical "clues" and is thus more faithful to the teacher's decision-making process.

Results and Key Findings

Experiments on TripAdvisor and Amazon review datasets revealed a staggering "inconsistency crisis," with 62% to 86% of base model explanations (Mistral, Llama-2, Yi) being inconsistent. Applying PEX-DPO led to:

  • Massive Consistency Gains: PEX scores improved by 43.1% to 292.3% across model families.
  • Faithfulness Boost: Student models were 1.5% to 9.7% better at predicting teacher labels when provided with consistency-optimized explanations.
  • Robustness: The PEX metric proved robust to prompt formatting, capturing fundamental model properties.

Conclusion and Limitations

While consistency is not a guarantee of faithfulness, this research demonstrates it is a necessary prerequisite. The current PEX framework is limited to binary classification and requires access to model log-probabilities, hindering its use on black-box APIs. Future work will explore multi-class extensions and application to Chain-of-Thought reasoning, where explanations precede predictions.

Summary updated

BibTeX
@misc{zhao2025necessary,
  author = {Lingjun Zhao and Hal Daumé},
  title = {{A Necessary Step toward Faithfulness: Measuring and Improving Consistency in Free-Text Explanations}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.797},
}
Figure