Properties and Challenges of LLM-Generated Explanations

Jenny Kunz, Marco KuhlmannNAACL 2024

TL;DR

Large Language Models (LLMs) are increasingly generating explanations for their outputs, mimicking human explanation styles. This paper investigates whether these LLM-generated explanations exhibit common human explanation properties, finding they often do, which can be both beneficial and detrimental depending on the user's goals.

What Problem Does This Paper Address?

As LLMs become more capable, they are being used in applications where understanding why a model made a certain decision is crucial. While previous research focused on explanations from models trained on specific, well-defined tasks, current LLMs can generate explanations for a much broader range of inputs. However, LLMs are not explicitly trained on annotated explanations; instead, they learn from vast amounts of human-generated text. This raises a key question: what properties do these LLM-generated explanations possess, and how do these properties align with the goals of explainable AI (XAI), which often prioritize faithfulness and transparency? The paper aims to bridge this gap by analyzing the characteristics of explanations generated by LLMs.

Methods

The researchers adopted a human evaluation approach. They used a subset of the Alpaca dataset, which consists of instructions and outputs generated by GPT-4. They then designed a questionnaire with six questions to assess specific properties of the explanations. These properties, derived from existing literature on human explanations, include:

  • Presence of Explanation: Does the output actually contain an explanation?
  • Selectivity: Does the explanation focus on a subset of relevant factors?
  • Subjectivity: Does the explanation include personal opinions or biases?
  • Illustrative Elements: Does the explanation use examples to clarify?
  • Misleading Explanations: Is the explanation factually incorrect or deceptive?

Three annotators, all experts in LLMs, evaluated 200 instruction-output pairs. They also answered a question about whether they would provide an explanation for the same prompt, allowing for a comparison between LLM-generated explanations and human intent.

Datasets

The primary dataset used in this study was Alpaca. This dataset was automatically generated using the self-instruct pipeline with GPT-4 annotations. The researchers filtered this dataset to select 200 instructions that they believed would benefit from an explanation for their primary output.

Key Insights and Ideas

  • LLMs Mimic Human Explanations: The study strongly suggests that LLMs, due to their pre-training on human text, adopt common properties of human explanations, even when not explicitly trained to do so.
  • Selectivity and Illustration are Common: The most prevalent properties observed in LLM-generated explanations were selectivity (focusing on a subset of factors) and the inclusion of illustrative elements (like examples). This aligns with how humans often simplify complex information.
  • Subjectivity and Misleading Explanations are Less Common: Subjective explanations were rare, likely due to alignment techniques in LLMs (like RLHF) that penalize bias. Misleading explanations were also infrequent in the Alpaca dataset, though the authors hypothesize this might be because Alpaca's instructions are high-probability and thus less prone to incorrect answers.
  • Explanation Properties Vary by Task: The study found that the prevalence of explanation properties differs across task categories (e.g., math vs. coding vs. factual questions), indicating that a one-size-fits-all approach to explanations is not ideal.
  • Tension Between XAI Goals: There's an inherent tension between the goals of XAI (faithfulness, transparency) and the communicative function of human-like explanations (understandability, persuasiveness). LLM explanations often lean towards the latter.

Key Results

  • Explanation Presence: GPT-4 provided explanations for approximately 64.3% of the outputs on average, with a notable variation among annotators.
  • Selectivity: This property was present in 61 out of 137 explanations where at least two raters agreed an explanation existed.
  • Illustrative Elements: These were found in 58 explanations.
  • Subjectivity: Only 8 explanations were rated as subjective.
  • Misleading Explanations: Just 1 explanation was flagged as misleading.
  • Category Differences: Explanations for "List/Facts" were the most common and verbose. "Grammar & Language" and "Text Classification" tasks were least likely to be explained. Math and code questions showed a moderate rate of explanation.

Limitations and Future Work

The authors acknowledge several limitations:

  • Annotation Method: Asking annotators if they would explain is not a direct measure of actual explanation behavior.
  • Dataset Bias: The Alpaca dataset, being LLM-generated, likely contains instructions that lead to correct answers, potentially reducing the occurrence of misleading explanations for incorrect labels. This might not generalize to less common or more challenging prompts.
  • Generalizability: Findings might not generalize to low-resource domains or different instruction types. The use of GPT-4, a proprietary model, limits insights into open-source alternatives.
  • Cultural and Demographic Bias: The study was limited to English instructions and a relatively homogenous group of annotators, potentially reflecting cultural norms.

Future work could involve using a more diverse dataset, exploring explanations for incorrect labels more thoroughly, and investigating the impact of these properties on different user groups and XAI goals.

Summary updated

BibTeX
@inproceedings{kunz2024properties,
  author = {Jenny Kunz and Marco Kuhlmann},
  title = {{Properties and Challenges of LLM-Generated Explanations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure