TL;DR
This paper investigates a subtle but critical failure mode in Large Language Models (LLMs) for question answering: "spurious" correct answers. They found that a significant portion of seemingly correct answers are derived from incorrect or hallucinated evidence, highlighting a gap between surface-level accuracy and true understanding. The research proposes methods to analyze and potentially mitigate this issue by leveraging external knowledge bases.
What Problem Does This Paper Address?
Large Language Models (LLMs) have achieved remarkable performance on complex tasks like Question Answering (QA). However, simply getting the right answer doesn't guarantee the LLM truly "understands" the underlying knowledge or reasoning process. This paper tackles the problem of "spurious" correct answers, where an LLM provides the correct final answer but relies on flawed or fabricated intermediate reasoning steps (evidence). This is crucial because it means LLMs might be succeeding through luck or by mimicking patterns rather than genuine knowledge retrieval and inference. Understanding the extent and nature of these spurious answers is essential for building more reliable and trustworthy AI systems.
Methods
The core of the paper's methodology involves analyzing the "derivation triples" – structured pieces of evidence that LLMs generate to justify their answers. They use a Chain-of-Thought (CoT) prompting strategy with GPT-4, which encourages the model to output its reasoning steps alongside the final answer. This allows researchers to examine the intermediate knowledge used by the LLM.
Key aspects of their method include:
- Multi-hop QA Datasets: They utilize datasets (HotPotQA, 2WikiMultiHopQA, JEMHopQA) that require combining information from multiple sources to answer a question, making them ideal for analyzing multi-step reasoning.
- Derivation Triple Evaluation: They manually and automatically evaluate the correctness of the generated derivation triples against "gold" (human-curated) derivations.
- Spurious Answer Identification: They define spurious correct answers as cases where the final answer is correct (T) but the derivation triples are incorrect (F).
- External Knowledge Base Integration: They explore how combining LLM-generated evidence with existing knowledge bases like Wikidata and Shinra can help correct or complement flawed reasoning.
Datasets
The study uses the following multi-hop QA datasets:
- HotPotQA: A Wikipedia-based dataset for English QA.
- 2WikiMultiHopQA: Another Wikipedia-based dataset for English QA.
- JEMHopQA: A Wikipedia-based dataset for Japanese QA.
They also created focused datasets for numerical comparison questions:
- HotPot_NC, 2Wiki_NC, JEMHop_NC: Subsets of the above datasets focusing on numerical comparison questions.
- DROP_NC: A dataset derived from DROP, specifically for numerical comparison questions requiring mathematical operations.
Key Insights and Ideas
- Ubiquitous Spuriousness: A significant percentage of LLM answers, even correct ones, are "spurious," meaning they are not grounded in correct evidence. This suggests LLMs are not always reasoning from facts.
- Numerical Comparisons are Vulnerable: Questions involving numerical comparisons (dates, values) are particularly prone to spurious correct answers. This is because the relative order of numbers might remain correct even if the numbers themselves are slightly off.
- Compositional Questions are Harder: Questions requiring the identification of "bridge entities" (entities that link two pieces of information) are more challenging and lead to more errors.
- External KBs are Crucial: Combining LLM-generated evidence with structured knowledge bases can significantly improve the coverage and correctness of the reasoning, offering a path to mitigate hallucinations.
- LLM Reasoning is Not Always Causal: The fact that correct answers can arise from incorrect derivations raises questions about the causal relationship between the LLM's internal knowledge representation and its output.
Key Results
- Up to 31% of correct answers are spurious across the evaluated datasets. This rate can be as high as 40% for numerical comparison questions.
- GPT-4, even with advanced prompting like CoT, exhibits these spurious correct answers.
- Composition questions are significantly harder than comparison questions.
- Combining GPT-4's derivations with Wikidata and Shinra increased the coverage of gold derivation triples from 51% to 59% for HotPot and 48% to 63% for JEMHop.
Limitations and Future Work
- Assumption of Derivation Correctness: The study assumes that the derivation triples generated by the LLM represent its actual reasoning process. Future work could aim to verify this assumption.
- Manual Evaluation of Derivations: While automated metrics are used for answers, derivation triples often require manual evaluation, which can be time-consuming and subjective.
- Focus on Specific LLM: The study primarily uses GPT-4. Further research could explore whether these findings generalize to other LLMs.
- Mitigation Strategies: While external KBs show promise, more sophisticated methods for integrating LLM reasoning with KBs are needed. Investigating how to precisely "fix" the numerical aspect of grounding information is also highlighted as future work.
Summary updated