TL;DR
This paper investigates how well large language models (LLMs) can extract relevant evidence from text to justify their predictions in few-shot learning scenarios. They find that while LLMs are generally good at quoting evidence, errors in prediction are more often due to misinterpreting correct evidence rather than failing to find it, which is a promising sign for building reliable AI systems.
What Problem Does This Paper Address?
Ensuring that AI systems, particularly large language models (LLMs), align with human intentions and values is a critical challenge in AI alignment. A key aspect of this is "verification"—the ability for humans to check if an LLM's decision is sound. Traditionally, interpretability methods aimed to help with this by providing explanations. However, with the rise of LLMs that can generate free-text explanations (abstractive approaches), it's become harder to directly link these explanations to the original source text. This paper focuses on "extractive evidence"—literally quoting snippets from the input document—as a more verifiable form of explanation. The core problem is understanding whether LLMs can reliably retrieve and use this extractive evidence to support their predictions, and crucially, what kinds of errors they make when they fail. This is vital for building human-AI collaboration systems where humans can trust and audit LLM outputs.
Methods
The researchers explored the ability of LLMs to perform "self-rationalization," where the model generates its own extractive explanations. They tested two prominent proprietary LLMs: GPT-4 and Gemini. Their experimental setup involved various prompting conditions to isolate different aspects of the process:
- Predict-then-explain vs. Explain-then-predict: Investigating if the order of generating a label and then evidence, or vice-versa, affects performance.
- Evidence given vs. Evidence occluded: Comparing performance when human-annotated evidence is provided directly to the model versus when it's removed from the document.
- Zero-shot vs. Few-shot learning: Evaluating performance with and without providing a few examples.
They measured "evidence quoting failure rates" (how often the model fails to extract any relevant quote) and "label accuracy" across different conditions. Crucially, they analyzed the types of errors made, distinguishing between failing to retrieve evidence and misinterpreting retrieved evidence.
Datasets
The study utilized five datasets with gold-standard human-annotated extractive evidence:
- MultiRC (MRC): Reading comprehension with questions requiring information from multiple sentences.
- SciFact (SF): Scientific claim verification (support, refute, or no info).
- WikiAttack (WA): Identifying personal attacks in Wikipedia comments.
- Evidence Inference (EI): Comparing the effectiveness of two treatments for a given outcome.
- HealthFC (HFC): Medical domain QA dataset for health-related claims.
For cost efficiency, they used randomly sampled 300-item subsets of the development set for each dataset.
Key Insights and Ideas
- Extractive evidence is key for verification: Abstractive explanations can obscure the link between the input and the prediction, making verification difficult. Extractive evidence, by contrast, directly grounds the prediction in the source text.
- LLMs are good at quoting, but not perfect: While generally reliable, some datasets (like Evidence Inference) showed higher failure rates in evidence quoting, possibly due to longer context lengths.
- Misinterpretation > Missing Evidence: For most datasets, prediction errors were more often caused by the LLM misinterpreting correctly retrieved evidence or being misled by confounding evidence, rather than failing to find any relevant evidence at all. This is a hopeful sign, as misinterpretation is more amenable to correction through prompting or human review than a complete lack of evidence.
- Order of operations matters (sometimes): The "predict-then-explain" order sometimes led to slightly lower label accuracy compared to "explain-then-predict," suggesting that forcing a justification upfront can be a minor hindrance.
- Evidence quality is crucial: The study highlights that the nature of the evidence and the document complexity significantly impact LLM performance.
Key Results
- Low evidence quoting failure rates: Across most datasets and models, the rate at which LLMs failed to quote any evidence was generally low (e.g., <5% for many cases).
- Self-rationalization impact: Forcing LLMs to self-rationalize (provide evidence) had a substantial positive effect on accuracy for Evidence Inference and HealthFC, but a marginal or even negative effect on other datasets.
- Error attribution: For MultiRC, SciFact, and WikiAttack, prediction errors were predominantly linked to misinterpreting evidence (confounding or uninterpretable evidence) rather than missing key evidence. For HealthFC, missing evidence was a more significant factor.
- Evidence retrieval vs. interpretation: When models missed evidence, it was often because there was other viable evidence they could use, suggesting they weren't truly "missing" it but rather failing to prioritize or correctly identify it in the context of other information.
Limitations and Future Work
- Dataset-specific performance: The effectiveness of extractive self-rationalization varies significantly across datasets, likely due to document length and evidence complexity.
- Proprietary models: The study focused on GPT-4 and Gemini, and findings might differ for other models.
- Future work: The authors suggest developing evaluation protocols to identify datasets where extractive self-rationalization is most likely to be effective. Further research could explore how to better support evidence retrieval with additional context and how to improve the interpretation of retrieved evidence. The findings also point towards the need for systems that can help humans identify and correct misinterpretations of evidence.
Summary updated