Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models' Explanations (Student Abstract)

Mu-Tien Kuo, Chih-Chung Hsueh, Richard Tzong-Han TsaiAAAI 2024 · DOI

TL;DR

Large Language Models (LLMs) are increasingly used for tasks requiring explanations, but evaluating these explanations is challenging. This paper introduces a novel, task-agnostic framework to automatically assess the "fidelity" (truthfulness to the model's reasoning) and "interpretability" (human understandability) of LLM explanations, overcoming limitations of previous methods that required full model access or costly human evaluation.

What Problem Does This Paper Address?

As LLMs like GPT-3.5 become more prevalent, their ability to generate explanations is crucial for trust and adoption. However, rigorously evaluating these explanations is a significant hurdle. Traditional methods for assessing explanation quality often rely on direct access to model internals (e.g., gradients, internal features) or human annotators. These approaches are problematic for proprietary LLMs where such access is restricted. Furthermore, human evaluation is expensive, time-consuming, and prone to subjectivity. This paper addresses the need for an automated, task-agnostic framework that can evaluate LLM explanations without requiring deep model introspection or human judgment, making the evaluation process more scalable and practical.

Methods

The authors propose a two-pronged evaluation framework:

  • Fidelity: Instead of relying on internal model access, they employ a "textual perturbation" approach. This involves subtly altering the input text and observing how consistently the LLM's explanation reflects its decision. If the explanation remains stable across these minor input changes, it suggests higher fidelity, meaning the explanation accurately reflects the model's underlying reasoning.
  • Interpretability: To avoid the subjectivity and cost of human evaluators, they leverage other Large Language Models (LLMs) to assess interpretability. This automated approach breaks down interpretability into three cognitive dimensions:
  • Rationale Coherence (RC): Assesses the logical consistency of sentences within an explanation using Natural Language Inference (NLI) models to detect contradictions.
  • Cognitive Fluency (CF): Measures how well an explanation's logic aligns with general human thought patterns, evaluated by LLMs trained on tasks like reading comprehension.
  • Cognitive Salience (CS): Evaluates the readability and complexity of the explanation's language using transformer-based encoders.

Datasets

The paper mentions using the SemEval-2016 Task 6 dataset for their experiments. They also refer to the DROP dataset in the context of evaluating LLMs for human-like rationale understanding.

Key Insights and Ideas

  • Automated Interpretability is Feasible: Using LLMs to evaluate interpretability offers a promising alternative to human annotators, addressing scalability and subjectivity concerns.
  • Textual Perturbations for Fidelity: A clever workaround for proprietary LLMs, this method allows for fidelity assessment without needing to peek inside the model's black box.
  • Prompt Engineering Matters (and can be tricky): The study highlights how different prompts (e.g., Chain-of-Thought vs. a "faithful" prompt) can impact explanation quality, sometimes leading to unexpected trade-offs.
  • Faithfulness vs. Fluency Trade-off: The authors observed that prompts designed to elicit "faithful" explanations might inadvertently lead to explanations that are less accurate to the model's true reasoning, suggesting a potential misunderstanding of "faithfulness" by the LLM itself.

Key Results

The paper presents results from evaluating GPT-3.5 on the Stance Detection task. Key findings include:

  • GPT-3.5's explanations showed moderate fidelity and satisfactory interpretability scores.
  • CoT (Chain-of-Thought) prompts generally yielded more accurate explanations, but sometimes included irrelevant intermediate steps.
  • F-CoT (Faithful Chain-of-Thought) prompts resulted in more cohesive arguments but at the expense of faithfulness, indicating GPT-3.5's potential misinterpretation of the "faithful" instruction.
  • Table 1 provides quantitative scores for Fidelity (F), Utility (U), Rationale Coherence (RC), Cognitive Fluency (CF), and Cognitive Salience (CS) across different prompt configurations (0-shot/4-shot, CoT/F-COT). For example, 0-shot CoT achieved an F score of 0.61 and a CF score of 0.89.

Limitations and Future Work

  • Limitations: The authors acknowledge that their framework, while automated, still relies on the capabilities of the LLMs used for evaluation. The observed trade-off with F-CoT prompts suggests that LLMs might not fully grasp the nuances of "faithfulness."
  • Future Work: The paper implicitly suggests further research into refining prompt engineering for better fidelity and exploring more sophisticated LLM-based evaluation metrics. The framework itself is designed to be adaptable and could be applied to a wider range of tasks and LLMs.

Summary updated

BibTeX
@inproceedings{kuo2024automated,
  author = {Mu-Tien Kuo and Chih-Chung Hsueh and Richard Tzong-Han Tsai},
  title = {{Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models' Explanations (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30470},
}
Figure