SAC: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley A. Malin, Kumar Sricharan

TL;DR

While checking if an LLM's answers are "self-consistent" is a popular way to detect hallucinations in black-box models, this paper proves that models can be consistently wrong. To fix this, the authors introduce SAC3, a method that cross-checks answers by subtly perturbing the questions and validating responses across different language models.

What Problem Does This Paper Address?

Detecting when a Large Language Model (LLM) is hallucinating is notoriously difficult, especially for commercial "black-box" models like ChatGPT or Claude, which don't expose their internal token probabilities. To bypass this, state-of-the-art techniques rely on self-consistency: they sample multiple answers to the same question at a high temperature and see if they match.

However, the authors uncover two fatal flaws in relying solely on self-consistency:

  1. Question-level hallucinations: LLMs can be confidently and consistently wrong. If you ask an LLM the same question five times, it might give you the exact same incorrect answer every single time.
  2. Sampling-level noise: Conversely, a model might provide a perfectly factual answer at temperature 0, but generate varied, incorrect answers when sampled at higher temperatures, triggering a false positive for hallucination.
Figure 1: Key observation: solely checking the self-consistency of LLMs is not sufficient for deciding factuality. Left: generated responses to the same question may be consistent but non-factual. Right: generated responses may be inconsistent with the original answer that is factually correct.

Methods

To overcome these limitations, the authors propose SAC3 (Semantic-aware Cross-check Consistency), a three-stage framework that introduces question and model diversity:

  1. Question-level Cross-checking (Perturbation): Instead of asking the exact same question repeatedly, SAC3 uses an LLM to generate $k$ semantically equivalent variations of the prompt (e.g., changing "Is 3691 a prime number?" to "Can 3691 only be divided by 1 and 3691?").
  2. Model-level Cross-checking (Verification): The framework introduces a separate, often smaller, open-source "verifier" LLM (like Falcon-7b or Guanaco-33b) to answer the same perturbed questions.
  3. Semantic-aware QA-Pair Consistency Check: Rather than just comparing the raw text of the answers, SAC3 evaluates the consistency of the Question-Answer pairs as a whole. It uses an LLM to judge whether different QA pairs express the same semantic meaning, outputting a final combined hallucination score.
Figure 2: Overview of the proposed semantic-aware cross-check consistency (SAC3) method.

Datasets

The authors evaluated SAC3 across two distinct types of tasks:

  • Binary Classification QA:
  • Prime number: 500 questions checking the primality of numbers between 1,000 and 20,000.
  • Senator search: 500 template-based questions tracking US senators and their alma maters.
  • Open-Domain Generation QA:
  • HotpotQA-halu: 250 randomly sampled complex, multi-hop questions with manually annotated factuality labels for the LLM-generated answers.
  • NQ-open-halu: 250 questions from the Natural Questions benchmark, similarly processed and manually annotated to provide a challenging open-domain testbed.

Key Insights and Ideas

  • Consistency $\neq$ Factuality: The core takeaway is a cautionary tale for LLM evaluation—just because a model is certain and repetitive does not mean it is right.
  • Rephrasing Breaks the Loop: Confidently wrong answers often stem from question ambiguity or specific token triggers. By altering the syntax while preserving the semantics, SAC3 forces the model out of its deterministic hallucination loop.
  • Cross-Model Complementarity: Surprisingly, smaller open-source models can act as highly effective verifiers for larger proprietary models. They don't share the same training data biases, meaning they rarely hallucinate in the exact same way.

Key Results

SAC3 significantly outperformed standard self-consistency metrics ($SC^2$) across the board:

  • Classification Tasks: On balanced datasets, SAC3 achieved an outstanding 99.4% AUROC on the Prime Number dataset and 99.7% on Senator Search, compared to $SC^2$'s mediocre performance of 65.9% and 56.1% respectively.
  • Generation Tasks: In the trickier open-domain generation benchmarks, SAC3-all (using Guanaco-33b as a verifier) achieved an AUROC of 87.0% on HotpotQA (+12.8% over baseline) and 77.2% on NQ-open (+6.7% over baseline).

Limitations and Future Work

While powerful, SAC3 does introduce some trade-offs:

  • Computational Cost: Generating question perturbations and querying multiple models increases API calls and latency. The authors note that using 2–4 perturbations can mitigate this, as performance gains tend to plateau after 5 samples.
  • Task Scope: The current framework is strictly evaluated on Question-Answering datasets. The authors note that more research is required to see how well semantic cross-checking generalizes to long-form text generation, abstractive summarization, or complex multi-turn dialogues.

Summary updated

Abstract

Hallucination detection is a critical step toward understanding the trustworthiness of modern language models (LMs). To achieve this goal, we re-examine existing detection approaches based on the self-consistency of LMs and uncover two types of hallucinations resulting from 1) question-level and 2) model-level, which cannot be effectively identified through self-consistency check alone. Building upon this discovery, we propose a novel sampling-based method, i.e., semantic-aware cross-check consistency (SAC3) that expands on the principle of self-consistency checking. Our SAC3 approach incorporates additional mechanisms to detect both question-level and model-level hallucinations by leveraging advances including semantically equivalent question perturbation and cross-model response consistency checking. Through extensive and systematic empirical analysis, we demonstrate that SAC3 outperforms the state of the art in detecting both non-factual and factual statements across multiple question-answering and open-domain generation benchmarks.
BibTeX
@inproceedings{zhang2023sac,
  author = {Jiaxin Zhang and Zhuohang Li and Kamalika Das and Bradley A. Malin and Kumar Sricharan},
  title = {{SAC: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.1032},
}
Figure