SAC³: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A. Malin, Kumar Sricharan

TL;DR

This paper introduces SAC³, a novel method for detecting hallucinations in black-box language models. SAC³ goes beyond simple self-consistency checks by using semantically equivalent question perturbations and cross-model consistency checks to identify both question-level and model-level hallucinations, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Modern large language models (LLMs) are incredibly capable but prone to "hallucinations" – confidently asserting incorrect information. This is a major hurdle for their deployment in critical applications where factual accuracy is paramount. While self-consistency (whether an LLM gives similar answers to the same question) has been used as a proxy for factuality, this paper argues it's insufficient. They identify two key failure modes: 1) question-level hallucinations, where an LLM consistently answers a specific question incorrectly, and 2) model-level hallucinations, where different LLMs disagree on the factual answer. Existing methods often struggle with these, especially when only API access to LLMs is available, limiting the ability to inspect internal workings like token probabilities.

Methods

The core of the proposed method, SAC³ (Semantic-aware Cross-check Consistency), is built upon the idea of consistency but expands it significantly. It operates in three main stages:

  1. Question-level Cross-checking via Semantically Equivalent Perturbations: Instead of just asking the same question repeatedly, SAC³ generates semantically equivalent rephrased versions of the original question. This helps uncover question-level hallucinations by seeing if the LLM remains consistently wrong across different phrasings of the same underlying query.
  2. Model-level Cross-check with Additional Verifier LM: To address model-level hallucinations, SAC³ introduces a "verifier LM." It compares the responses of the target LLM with those of this verifier LM for the same set of perturbed questions. Discrepancies between the two models can signal hallucinations.
  3. Consistency Score Calculation: The method then calculates various consistency scores:
  4. Self-checking Consistency (SC²): A baseline measure of how consistent the target LLM is with itself across sampled responses.
  5. Question-level Consistency (SAC³-Q): Measures consistency across the semantically perturbed questions.
  6. Model-level Consistency (SAC³-M): Measures consistency between the target LM and the verifier LM for the original question.
  7. Cross-model Cross-question Consistency (SAC³-QM): Measures consistency between the target LM and the verifier LM across the perturbed questions.
  8. Final Score (SAC³-all): A weighted combination of these scores to provide a comprehensive hallucination detection metric.

Datasets

The paper evaluates SAC³ on two types of datasets:

  • Classification QA:
    • Prime number: A dataset of 500 questions about primality, with factual answers always "Yes" and synthesized hallucinations as "No."
    • Senator search: A dataset of 500 questions about US senators and their alma maters, with factual answers always "No" and synthesized hallucinations as "Yes."
  • Open-domain Generation QA:
    • HotpotQA-halu: 250 examples from HotpotQA with hallucinated answers generated by GPT-3.5-turbo, manually annotated for factuality.
  • NQ-open-halu: 250 examples from Natural Questions (NQ)-open, also with generated hallucinated answers and manual annotations.

Key Insights and Ideas

  • Self-consistency is not enough: The paper strongly demonstrates that LLMs can be consistently wrong. Simply checking if an LLM gives the same answer multiple times doesn't guarantee that answer is factual.
  • Question-level hallucinations are a distinct problem: LLMs can fail on specific questions, consistently providing incorrect information even when they are generally capable. Semantic perturbation is key to uncovering this.
  • Model-level differences matter: Different LLMs exhibit varying propensities for hallucination, highlighting the need for cross-model comparisons.
  • Verifier LMs add robustness: Using a second LLM as a "verifier" significantly improves the ability to detect hallucinations by providing an external check.
  • Flexibility in approach: The SAC³ framework is adaptable, allowing users to choose different LLMs for the target and verifier roles, and to tune parameters like the number of perturbed questions to balance accuracy and computational cost.

Key Results

  • Classification QA: SAC³-Q achieved AUROC scores above 99% on both balanced and unbalanced datasets, significantly outperforming the self-consistency baseline (SC²) which struggled, especially on unbalanced data (dropping to 48.2% and 29.6% accuracy). SAC³-all with Guanaco-33b achieved the highest accuracy (99.4%) on unbalanced classification tasks.
  • Open-domain Generation QA: SAC³-Q showed a +7% AUROC improvement over SC² on both HotpotQA-halu and NQ-open-halu. SAC³-all (with Guanaco-33b) achieved an AUROC of 87.0% and 77.2% respectively, outperforming SC² by a substantial margin.
  • Robustness: The paper shows that SAC³-Q and SAC³-all are more robust to threshold choices than SC² on unbalanced datasets.
  • Efficiency: The authors suggest that using 2-4 perturbed questions is often sufficient to achieve good performance with reduced computational cost.

Limitations and Future Work

  • Question-answering focus: The current framework is primarily evaluated on question-answering tasks. Future work should assess its generalizability to more complex tasks like conversational or dialogue-based prompting.
  • Semantic equivalence checks: The accuracy of semantic equivalence checks, especially with more complex prompts, could be further investigated.
  • Efficiency-utility trade-off: While the paper explores this, further research into optimizing sample sizes and prompt strategies for faster, more efficient detection is suggested.
  • Annotated datasets: The authors plan to release their annotated datasets to encourage further research.

Summary updated

Abstract

Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC3) that expands on the principle of self-consistency checking. Our SAC3 approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC3 outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.
BibTeX
@inproceedings{zhang2023sac,
  author = {Jiaxin Zhang and Zhuohang Li and Kamalika Das and Bradley A. Malin and Kumar Sricharan},
  title = {{SAC³: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.1032},
}
Figure