Abstract
Recent progress in natural language processing (NLP) owes much to remarkable advances in large language models (LLMs). Nevertheless, LLMs frequently "hallucinate," resulting in non-factual outputs. Our carefully-designed human evaluation substantiates the serious hallucination issue, revealing that even GPT-3.5 produces factual outputs less than 25% of the time. This underscores the importance of fact verifiers in order to measure and incentivize progress. Our systematic investigation affirms that LLMs can be repurposed as effective fact verifiers with strong correlations with human judgments. Surprisingly, FLAN-T5-11B, the least factual generator in our study, performs the best as a fact verifier, even outperforming more capable LLMs like GPT3.5 and ChatGPT. Delving deeper, we analyze the reliance of these LLMs on high-quality evidence, as well as their deficiencies in robustness and generalization ability. Our study presents insights for developing trustworthy generation models.
Introduction: The Hallucination Crisis
The "hallucination" problem is the Achilles' heel of modern NLP. LLMs frequently generate outputs that are fluent and persuasive but factually incorrect. This paper tackles two primary questions: how bad is the hallucination problem in state-of-the-art models, and can we use these same models to fix the problem by repurposing them as automated fact-checkers?
Through rigorous human evaluation, the authors quantified the lack of factuality in models like FLAN-T5, Llama, and GPT-3.5. The results were sobering. As shown in the data, even the most advanced models struggle to maintain factuality, especially when generating longer paragraphs.
Methodology: The Retrieve-then-Verify Framework
To turn an LLM into a verifier, the researchers employed a "Retrieve-then-Verify" framework. Given a statement, the process follows these steps:
- Retrieval: The Contriever model is used to find relevant passages from Wikipedia to serve as external evidence.
- Prompting: The retrieved evidence and the statement are fed into an LLM with a simple prompt: "Is the statement correct? (A) Yes. (B) No."
- Scoring: Instead of relying on text output, the authors normalized the model's log-probabilities (logits) for the tokens "Yes" and "No" to produce a continuous factuality score. This allows for more nuanced ranking than a binary label.
- De-contextualization: For paragraph-level verification, ChatGPT was used to perform coreference resolution (e.g., replacing pronouns with proper nouns) to make individual sentences easier to verify in isolation.
Key Insights and Results
The "Underdog" Winner
The most surprising finding was the clear asymmetry in model capability, termed the Generation-Verification Gap. FLAN-T5 (11B)—the model that performed the worst at generating factual text—was the best at verifying it. It consistently outperformed GPT-3.5 and ChatGPT in verification accuracy and calibration.
Knowledge vs. Evidence
A model might not "know" a fact well enough to retrieve it from its weights during generation, but it can "recognize" the truth when the evidence is placed right in front of it. However, the study found that ChatGPT tends to rely more on its internal "memorized" knowledge. While this helps it handle "relevant but fake" evidence better than other models, it also makes it more susceptible to ignoring provided evidence in favor of its own internal biases.
Numerical Weakness
Despite their strengths, all models struggled with "Cardinal" and "Ordinal" entity types. LLMs still find it difficult to reason over inter-numeral logical relations (e.g., confusing "36 years old" with "in his 40s"), indicating a persistent weakness in mathematical and logical reasoning within linguistic contexts.
Conclusion
This study demonstrates that while LLMs are prone to hallucination, they possess a latent ability to verify facts that exceeds their ability to generate them. By utilizing external evidence and log-probability scoring, even smaller instruction-tuned models like FLAN-T5 can serve as robust fact-checkers. Future work should focus on autonomous reasoning agents that can interactively search for and verify information through multiple steps to further bridge the gap between fluency and factuality.
Summary updated