TL;DR
This paper introduces SAC$^3$, a novel method for detecting hallucinations in black-box language models (LMs) by going beyond simple self-consistency checks. SAC$^3$ leverages semantically equivalent question perturbations and cross-model consistency to identify both question-specific and model-specific hallucinations, achieving state-of-the-art performance on various QA and generation tasks.
What Problem Does This Paper Address?
Modern large language models (LLMs) are incredibly capable but often "hallucinate" – they confidently generate factually incorrect information. This is a major hurdle for their deployment in critical applications where accuracy is paramount. Detecting these hallucinations is crucial for building trust in LLMs. Existing methods often rely on "self-consistency," assuming that if an LM generates similar answers to the same question multiple times, those answers are likely factual. However, this paper argues that this assumption is flawed. LLMs can consistently produce incorrect answers (question-level hallucinations) or exhibit differing hallucination tendencies across different models (model-level hallucinations), both of which can evade simple self-consistency checks. Furthermore, many powerful LLMs are only accessible via APIs, meaning researchers can't inspect their internal workings (like token-level probabilities) to assess uncertainty directly.
Methods
The core of SAC$^3$ is its "semantic-aware cross-check consistency" approach, which builds upon self-consistency but adds crucial layers of verification. It operates in three main stages:
- Question-level Cross-checking via Semantically Equivalent Perturbations: Instead of just asking the same question repeatedly, SAC$^3$ generates semantically equivalent rephrased versions of the original question. This is done by prompting an LLM to provide alternative phrasings. The idea is that if an LM consistently hallucinates on a question, it's likely to do so even when the question is phrased slightly differently, but in a way that preserves the original meaning. This helps uncover question-level hallucinations.
- Model-level Cross-check with Additional Verifier LM: To address model-level hallucinations, SAC$^3$ introduces a "verifier LM." This is a separate, often smaller or differently trained, LLM. The approach then compares the responses of the target LM (the one being evaluated) with the responses of the verifier LM to the same set of original and perturbed questions. Discrepancies between the two models can highlight instances where one model is hallucinating while the other is not.
- Consistency Score Calculation: Finally, SAC$^3$ combines the insights from these cross-checks into a comprehensive score. It calculates:
- Self-checking Consistency (SC$^2$) Score: A baseline measure of consistency for the target LM's own responses.
- Question-level Consistency (SAC$^3$-Q) Score: Measures consistency across the target LM's responses to the original and semantically perturbed questions.
- Model-level Consistency (SAC$^3$-M) Score: Measures consistency between the target LM and the verifier LM on the original question.
- Model-level Cross-Question Consistency (SAC$^3$-QM) Score: Measures consistency between the target LM and the verifier LM across both original and perturbed questions. A final score, SAC$^3$-all, is a weighted combination of these scores, allowing for flexibility in how much weight is given to the verifier LM.
Datasets
The paper evaluates SAC$^3$ on two types of datasets:
- Classification QA:
- Prime number: A dataset with 500 questions about the primality of numbers, where factual answers are always "Yes" and synthesized hallucinations are "No."
- Senator search: A dataset with 500 questions about US senators and their alma maters, where factual answers are "No" and synthesized hallucinations are "Yes."
- Open-domain Generation QA:
- HotpotQA-halu: 250 examples from HotpotQA where answers were generated by GPT-3.5-turbo and then manually annotated for factuality.
- NQ-open-halu: 250 examples from Natural Questions (NQ)-open, similarly generated and manually annotated.
The paper notes that for classification tasks, they use synthesized hallucinated answers, while for generation tasks, they use LLM-generated answers that are then manually annotated.
Key Insights and Ideas
- Self-consistency is not enough: The paper strongly demonstrates that simply checking if an LM gives the same answer multiple times is insufficient for detecting hallucinations. LLMs can be consistently wrong.
- Question-level hallucinations exist and are detectable: LMs can consistently fail on specific questions, even if they are generally good. Perturbing questions semantically helps uncover these persistent errors.
- Model-level disparities are real: Different LLMs exhibit different tendencies to hallucinate. Using a verifier LM can help identify when a target LM is hallucinating while another model is not.
- Semantic equivalence is key: The method's success hinges on generating and checking semantically equivalent questions, not just superficial variations.
- Black-box accessibility is a constraint: The proposed method is designed to work with models accessible only via APIs, avoiding the need for internal model access.
Key Results
- Classification QA: SAC$^3$-Q significantly outperforms the self-consistency baseline (SC$^2$) with AUROC scores above 99% on balanced datasets. On unbalanced datasets (100% hallucinations), SAC$^3$-Q achieves accuracies of 93.2% and 97.0% for Prime Number and Senator Search, respectively, compared to SC$^2$'s 48.2% and 29.6%. SAC$^3$-all with Guanaco-33b achieves the highest accuracy (99.4%) on the Prime Number dataset.
- Open-domain Generation QA: SAC$^3$-Q shows better AUROC than SC$^2$ by 7% on both HotpotQA-halu and NQ-open-halu. SAC$^3$-all with Guanaco-33b achieves AUROC scores of 87.0% and 77.2% on these datasets, respectively.
- Robustness: The method is shown to be robust to threshold choices, especially SAC$^3$-Q and SAC$^3$-all.
- Efficiency: The paper suggests that using 2-4 perturbed questions is sufficient for good performance, balancing accuracy with computational cost.
Limitations and Future Work
- Question-answering focus: The current framework is primarily evaluated on question-answering tasks. Future work should assess its generalizability to more complex tasks like conversational or dialogue-based prompting.
- Efficiency-utility trade-off: While the paper explores this, further investigation into optimizing sample sizes and prompt strategies for better efficiency is suggested.
- Semantic equivalence checks: The accuracy of semantic equivalence checks on more complex tasks needs further study.
- Computational cost: While parallelization is mentioned, further optimization of prompt strategies for semantic consistency checking could reduce complexity.
- Verifier LM selection: The paper notes that the choice of verifier LM can be task-dependent, suggesting further research into optimal selection strategies.
Summary updated