TL;DR
While checking if an LLM's answers are "self-consistent" is a popular way to detect hallucinations in black-box models, this paper proves that models can be consistently wrong. To fix this, the authors introduce SAC3, a method that cross-checks answers by subtly perturbing the questions and validating responses across different language models.
What Problem Does This Paper Address?
Detecting when a Large Language Model (LLM) is hallucinating is notoriously difficult, especially for commercial "black-box" models like ChatGPT or Claude, which don't expose their internal token probabilities. To bypass this, state-of-the-art techniques rely on self-consistency: they sample multiple answers to the same question at a high temperature and see if they match.
However, the authors uncover two fatal flaws in relying solely on self-consistency:
- Question-level hallucinations: LLMs can be confidently and consistently wrong. If you ask an LLM the same question five times, it might give you the exact same incorrect answer every single time.
- Sampling-level noise: Conversely, a model might provide a perfectly factual answer at temperature 0, but generate varied, incorrect answers when sampled at higher temperatures, triggering a false positive for hallucination.
Methods
To overcome these limitations, the authors propose SAC3 (Semantic-aware Cross-check Consistency), a three-stage framework that introduces question and model diversity:
- Question-level Cross-checking (Perturbation): Instead of asking the exact same question repeatedly, SAC3 uses an LLM to generate $k$ semantically equivalent variations of the prompt (e.g., changing "Is 3691 a prime number?" to "Can 3691 only be divided by 1 and 3691?").
- Model-level Cross-checking (Verification): The framework introduces a separate, often smaller, open-source "verifier" LLM (like Falcon-7b or Guanaco-33b) to answer the same perturbed questions.
- Semantic-aware QA-Pair Consistency Check: Rather than just comparing the raw text of the answers, SAC3 evaluates the consistency of the Question-Answer pairs as a whole. It uses an LLM to judge whether different QA pairs express the same semantic meaning, outputting a final combined hallucination score.
Datasets
The authors evaluated SAC3 across two distinct types of tasks:
- Binary Classification QA:
- Prime number: 500 questions checking the primality of numbers between 1,000 and 20,000.
- Senator search: 500 template-based questions tracking US senators and their alma maters.
- Open-Domain Generation QA:
- HotpotQA-halu: 250 randomly sampled complex, multi-hop questions with manually annotated factuality labels for the LLM-generated answers.
- NQ-open-halu: 250 questions from the Natural Questions benchmark, similarly processed and manually annotated to provide a challenging open-domain testbed.
Key Insights and Ideas
- Consistency $\neq$ Factuality: The core takeaway is a cautionary tale for LLM evaluation—just because a model is certain and repetitive does not mean it is right.
- Rephrasing Breaks the Loop: Confidently wrong answers often stem from question ambiguity or specific token triggers. By altering the syntax while preserving the semantics, SAC3 forces the model out of its deterministic hallucination loop.
- Cross-Model Complementarity: Surprisingly, smaller open-source models can act as highly effective verifiers for larger proprietary models. They don't share the same training data biases, meaning they rarely hallucinate in the exact same way.
Key Results
SAC3 significantly outperformed standard self-consistency metrics ($SC^2$) across the board:
- Classification Tasks: On balanced datasets, SAC3 achieved an outstanding 99.4% AUROC on the Prime Number dataset and 99.7% on Senator Search, compared to $SC^2$'s mediocre performance of 65.9% and 56.1% respectively.
- Generation Tasks: In the trickier open-domain generation benchmarks, SAC3-all (using Guanaco-33b as a verifier) achieved an AUROC of 87.0% on HotpotQA (+12.8% over baseline) and 77.2% on NQ-open (+6.7% over baseline).
Limitations and Future Work
While powerful, SAC3 does introduce some trade-offs:
- Computational Cost: Generating question perturbations and querying multiple models increases API calls and latency. The authors note that using 2–4 perturbations can mitigate this, as performance gains tend to plateau after 5 samples.
- Task Scope: The current framework is strictly evaluated on Question-Answering datasets. The authors note that more research is required to see how well semantic cross-checking generalizes to long-form text generation, abstractive summarization, or complex multi-turn dialogues.
Summary updated