Best-Worst Claim Comparisons for Zero-Source Hallucination Detection

When a language model writes a long answer, some of its sentences are true and some are made up, and in many deployed settings we cannot look up the truth or read the model's internal probabilities. This note proposes asking the model a different question. Instead of "how sure are you that this sentence is true?", it asks "which of these two sentences are you more sure of, and by how much?" It takes this protocol from multi-criteria decision-making, the field that turns human expert judgments into numerical weights. There, the Best-Worst Method recovers a consistent set of weights from a small number of comparisons against a "best" and a "worst" anchor, and it reports how self-contradictory those comparisons were. We use the model's own claims as the items to be weighted, and a known-true and a known-fabricated statement as the anchors. The result is a per-claim confidence score. It also yields a second signal that no current zero-source detector has: how inconsistent the model's comparative beliefs are within a single answer.

The bet has two parts. Comparative judgments, pinned to anchors, may separate true from fabricated claims better than absolute self-ratings. The inconsistency of those judgments may also flag answers the model is confabulating. Both parts can fail, and the experiments below are built to show quickly whether either one holds.

Core Claim

Decision analysts do not trust a single absolute rating from an expert. They collect comparisons and check them for internal consistency: if A is preferred to B by 3 and B to C by 3, then A should be preferred to C by roughly 9. We claim that a language model's beliefs about its own claims can be audited the same way, and that the violations are informative. For a long-form answer decomposed into atomic claims $c_1,\dots,c_n$, we add an anchor claim $c_B$ (a trivially true fact about the same entity, taken from the prompt) and an anchor claim $c_W$ (a deliberately fabricated fact). The model then rates, on a 1–9 scale, how much more confident it is in $c_B$ than in each $c_j$, and in each $c_j$ than in $c_W$. The Best-Worst Method, as used for criteria weighting by You and Yang, solves

$$\min_{w,\xi}\ \xi \quad \text{s.t.}\quad |w_B - a_{Bj} w_j| \le \xi w_j,\ \ |w_j - a_{jW} w_W| \le \xi w_W,\ \ \textstyle\sum_j w_j = 1,\ w \ge 0,$$

where $a_{Bj}$ is the model's stated ratio of confidence in the true anchor over claim $j$, $a_{jW}$ is its stated ratio of confidence in claim $j$ over the fabricated anchor, $w_j$ is the recovered confidence weight of claim $j$, and $\xi$ is the smallest slack that makes all stated ratios compatible.

The weight $w_j$, placed between $w_W$ and $w_B$, is the per-claim hallucination score. The optimal slack $\xi^*$ is an answer-level inconsistency score. The method needs $2n-3$ comparisons rather than $n(n-1)/2$, so its cost grows linearly in the number of claims. It needs only text in and text out, so it runs against closed APIs with no retrieval.

The new part is not "MCDM plus LLMs". It is the specific claim that anchored comparative elicitation and its consistency diagnostic carry hallucination signal that absolute verbalized confidence does not.

Relation to Prior Work

The closest work is Zero-source LLM Hallucination Detection with Human-like Criteria Probing, which we call HCPD. HCPD has an evaluator break a response into criteria such as temporal consistency, assign weights and 1–10 scores, and aggregate them. It trains that evaluator with GRPO, a reinforcement-learning method, under weak supervision, and reports an average AUROC of 88.19% across TriviaQA, SciQ, NQ Open and CoQA on Llama-3.1-8b. The obvious extension at this intersection is to replace HCPD's averaging with TOPSIS or VIKOR, and its free-form weights with AHP or BWM.

We deliberately do not do that. Once criterion scores exist, a logistic regression fit on a few hundred labeled examples will almost certainly aggregate them better than any fixed ranking formula. A hand-designed ranking method has no clear advantage over a learned one here.

What MCDM offers that learning cannot trivially replace is its elicitation protocol and its consistency check. We therefore move the MCDM machinery from criteria to claims, and from the aggregation step to the measurement step. HCPD also scores whole short answers. We target claim-level detection in long-form text, which HCPD itself names as open.

The paper Enhancing Adversarial Robustness of LLMs with Analytic Hierarchy Process uses AHP only as a metaphor for staged prompting against jailbreaks. It computes no comparison matrices or consistency ratios, and it does not address factuality. Our proposal uses the actual optimization and its consistency diagnostic.

Two papers motivate the design. Ayush Agrawal et al. show that inconsistency across indirect questions about a model's own references signals fabrication. Our $\xi^*$ is a structured, single-pass analogue: inconsistency among comparisons rather than variance across samples. FActScore supplies the atomic-claim decomposition and the labeled biographies. It also documents that precision collapses on rare entities, which is exactly where zero-source signals are most needed.

One paper constrains the design. HaluEval reports that showing a judge a factual and a hallucinated response side by side degraded its judgment, relative to judging each response alone. This is a direct threat to any comparative protocol, and we address it head on below. Our comparisons are between single claims and fixed anchors, not between two competing full responses. Whether that distinction matters is an empirical question the experiments answer.

Research Program

We first build the elicitation pipeline. Claims are extracted with the FActScore decomposition. Anchors are generated per entity: the true anchor is restated from the prompt, and the false anchor is a fabricated attribute checked to be absent from the entity's reference page. The model's 1–9 ratings are then collected either one comparison per call or in batched calls.

Long answers can contain 30 or more claims, so we split them into blocks of about seven claims that share the same anchors. We solve one BWM problem per block and normalize weights through the shared anchors. This keeps prompts short while keeping all scores on one scale.

Second, we treat $\xi^*$ as an answer-level signal. We ask whether it predicts an answer's fraction of unsupported claims beyond what the mean claim weight already predicts. We also ask whether claims whose removal most reduces $\xi^*$ are disproportionately false, which would give a second, consistency-based per-claim score.

Third, we test transfer to settings the file identifies as hard. These are rare entities, the later parts of long answers where FActScore observes accuracy decay, and HaluEval's summarization split, where direct judgment is near chance.

Baselines

The naive reference point is to predict every claim as supported. It has AUROC 0.5 by construction, and its accuracy equals the benchmark's support rate. For context, FActScore reports 58.3% factual precision for ChatGPT on biographies, so nearly half of that model's atomic claims are the positives a detector must find. A second naive point is monolithic prompting ("does this passage contain errors?"). For that, HaluEval reports ChatGPT accuracies of 62.59% on QA, 72.40% on dialogue and 58.53% on summarization.

The strongest simple competitor is absolute verbalized confidence. The same model, given the same claim blocks, rates each claim's probability of being true on a 0–100 scale, optionally with the same anchors shown as context. This is the baseline most likely to match us, because it uses identical information at lower cost. We run it with the same model, the same claim decomposition, the same token budget (we allow it extra repeated samples to spend any budget it saves), and the same number of prompt-tuning iterations on a shared development split.

The second competitor is multi-sample consistency in the spirit of Agrawal et al.. We sample several regenerations or indirect questions per claim and score how often the claim is reaffirmed. The number of samples is chosen so its total model calls equal our $2n-3$ comparisons.

The third is an HCPD-style criteria prober applied per claim. It uses prompting only, because we cannot reproduce HCPD's GRPO training from the file. Its reported 88.19% AUROC is answer-level on short QA. We found no result on Lacuna for claim-level zero-source detection on FActScore biographies, so no cited number is directly comparable to ours. To report against HCPD's own number, we also run our method on its four QA benchmarks, treating short answers as one or two claims. We expect little advantage there, because a consistency check among one or two claims has almost nothing to check.

Retrieval-based FActScore verification against Wikipedia serves as a non-zero-source ceiling, not a competitor.

Beating the field means a statistically reliable per-claim AUROC gain over absolute verbalized confidence at matched calls, on at least two evaluator models, with bootstrap confidence intervals over entities. Three ablations isolate the key ingredients: BWM without anchors (the scale then floats between answers), randomly chosen comparison pairs instead of anchor pairs, and weights with $\xi^*$ discarded.

Why not just…?

Why not just ask the model for an absolute confidence per claim? This is the real threat. It is cheaper, and HaluEval's contrast result could mean that comparisons hurt rather than help. Nothing in the file shows that comparative elicitation beats absolute elicitation for LLM judges. The matched-budget head-to-head is therefore the first experiment. If absolute confidence ties us on per-claim AUROC, the per-claim part of the proposal is dropped. What would remain is whether $\xi^*$ adds answer-level signal, because absolute ratings produce no analogous inconsistency measure.

Why not just sample multiple outputs and measure disagreement? Sampling works, as Agrawal et al. show, but Chakraborty et al. stress its latency cost. Our method uses a comparable number of calls, so the fair question is which use of those calls gives more signal per call. We report AUROC against number of model calls for both methods. We also test whether their errors are complementary, since a combination may be the practical winner.

Why not just retrieve and verify? When retrieval is available it should be used; PerplexityAI's 71.5% precision versus ChatGPT's 58.3% in FActScore shows its value. We keep it as a ceiling, and target the zero-source setting, where corpora are proprietary, absent, or thin on rare entities.

Why not just swap TOPSIS or VIKOR into HCPD? A few hundred labels and a logistic regression over HCPD's criterion scores would likely beat any fixed ranking formula. We will run exactly that learned aggregator as a sanity check. If it also beats our anchored weights once given the same labels, the elicitation idea has failed on its own terms.

Proposed Experiments

The main testbed is the FActScore biography set: 6,500 generations across 13 models, with human-verified atomic-fact labels. We evaluate Llama-3.1-8b, chosen for continuity with HCPD, plus one API model, in two conditions: judging their own generations, and judging other models' generations. Metrics are per-claim AUROC and calibration of the anchored weights. At the answer level, we measure the partial correlation of $\xi^*$ with the unsupported-claim rate, controlling for mean claim weight.

Results are broken down by entity frequency and by claim position in the answer. This tests whether the comparative signal survives where FActScore shows precision collapsing. Secondary tests apply claim decomposition to HaluEval's summarization and dialogue splits, and apply our method to HCPD's four QA benchmarks for a direct comparison with the reported 88.19%.

The contrast check replicates HaluEval's side-by-side versus isolated judgment on our own models. It then compares that effect with our claim-versus-anchor comparisons. Recovering the paradox for full responses while seeing no degradation for claim-anchor pairs would locate where the paradox comes from.

The decisive checks are three. The first is whether anchored comparisons beat absolute confidence per claim at matched budget. The second is whether $\xi^*$ carries answer-level information beyond mean confidence. The third is whether either advantage persists on rare entities. A negative result on the first two is also worth publishing. It would show that the consistency tools of decision analysis do not transfer to model self-assessment, which the file suggests has not been tested.