Language models often get a fact wrong in a specific way. Asked about a lesser-known physicist, the model gives the birth year, advisor, or prize of a different, better-known physicist from the same subfield, and it does so with full confidence. This note proposes a reference-free test for this failure. For each specific claim the model makes about a target entity, the model is also asked the same question about a few entities it is likely to confuse with the target, which we call lookalikes. A claim is suspect when two things hold: the attribute normally tells these entities apart, and the claimed value is exactly what the model believes about one of the lookalikes. The test borrows an idea from referring-expression generation, the NLG task of describing an object so a listener can pick it out from similar distractors. There, an attribute is judged by how well it separates the target from those distractors. We apply that judgment to checking what the model has already written.
This matters because the usual self-diagnosis signals are weakest on exactly this error. Token entropy and likelihood flag rare but true names as uncertain. Resampling consistency cannot catch a confusion that the model reproduces the same way every time. Lookalike probing asks a different question: is this value more characteristic of the target than of its nearest neighbors, in the model's own beliefs? The note specifies the detector, a mitigation step that backs off to a coarser claim only as far as the target stays distinguishable, and experiments on ERBench, HalluEntity, FActScore-style biographies and LongFact.
Core Claim
A large share of entity-level hallucinations on long-tail subjects are swaps: values that are true of, or believed by the model to be true of, a semantically close entity. The evidence that this is plausible is indirect but consistent. HalluEntity annotates 18,785 entities in GPT-3.5 biographies and finds error rates of 30.9% for proper nouns and 36.2% for numbers. The research file's synthesis of that benchmark lists conflating biographical details between similar entities as a recognized hallucination pattern. FActScore shows precision collapsing on rare entities. ERBench shows that errors on early hops compound in multi-hop relational questions.
Our claim is that a swap leaves a detectable trace. The model's belief about the lookalike's value on that attribute equals the value it asserted for the target. That trace is present even when the token-level confidence is high.
The scoring rule follows classical discriminative attribute selection. Consider a claim that entity $T$ has value $v$ for attribute $a$, and let $D(T)$ be a set of $k$ lookalikes. For each entity $e \in D(T) \cup \{T\}$, sample the model's answer to "what is $a$ of $e$?" several times. This gives an empirical belief $\hat p_e(v)$, the fraction of samples that return $v$. Following SLINT+, which weights attributes by how unique their values are across instances, we define the attribute's discriminability $\delta(a; D)$ as the normalized entropy of the model's modal answers across the lookalike set. It is near 1 when the lookalikes all have different values and near 0 when they share one (nationality, field). The swap score is
$$ s(T,a,v) = \delta(a; D)\cdot \max_{d \in D(T)} \hat p_d(v)\cdot \big(1 - \big[\hat p_T(v) - \max_{d} \hat p_d(v)\big]_+\big), $$
where $[x]_+ = \max(x,0)$.
In words, the score is high when three conditions hold together. The attribute usually separates these entities. Some lookalike is confidently believed to have $v$. The model's belief that the target has $v$ is not clearly stronger than its belief about that lookalike. The discriminability factor matters: it keeps the detector from penalizing true values that the target shares with its neighbors, which is the obvious failure of naively contrasting against a distractor.
The link to It's Not Polite to Point is direct. That paper showed that an attribute's discriminative value must be weighed against how reliably the observer can perceive it. Here the "observer" is the model's own memory of the lookalikes, and the sampled beliefs $\hat p_d$ play the role of classifier confidence.
Research Program
The work has three parts.
The first part is measuring how common swaps are. On ERBench, the database itself supplies both the lookalikes and the ground truth. Candidate lookalikes are rows sharing non-key attributes, for example movies with the same director and neighboring years. The functional dependencies tell us which attribute should discriminate, since Title, Year determines Director. A wrong answer counts as a swap if it equals the value of some neighboring row. On HalluEntity and FActScore biographies, we annotate each hallucinated entity for whether its value belongs to a close entity. Wikipedia is consulted for this labeling step only, never at test time.
This measurement is the go/no-go point. If swaps are a small minority of hallucinations, the detector's reach is small, and the project should be reframed as a targeted tool rather than a general detector.
The second part is the detector itself, which has two sources of lookalikes. The primary source is the model: it is prompted to list entities easily confused with $T$ that share its type and domain. The oracle source is a knowledge-base neighborhood, available on ERBench. Atomic claims are extracted as in FActScore. Each claim is parsed into (entity, attribute, value) and scored by $s$.
Cost is controlled in two ways. We probe only numbers and proper nouns, the categories HalluEntity identifies as most error-prone. We also cache each lookalike's beliefs across all claims about the same target.
The third part is discriminative backoff. Answer Only as Precisely as Justified replaces unsupported specific claims with coarser ones along a ladder of semantic granularity (exact date, then year, then decade, then omission). It does not ask whether the coarser claim still identifies the target. For a flagged claim, we choose the most specific rung at which the model's beliefs about the target and the lookalikes separate. For instance, "born in the early 1940s" survives if every lookalike was born outside that span. Otherwise the claim is omitted. We then calibrate the flag threshold on held-out data with the same overcommitment-aware utility objective that CSS uses. This keeps the two approaches comparable.
Baselines
The do-nothing reference is the unedited generation, scored by the same annotators or automatic verifier. For detection, it corresponds to random scoring, which gives an AUPRC equal to the base hallucination rate, 15% on HalluEntity.
The strongest existing reference-free detector is Focus, as evaluated in HalluEntity. Focus selects keywords and propagates uncertainty through the sequence. The file reports an AUROC of 0.78 and an AUPRC of 0.40 for Focus on Llama3-8B, the best in that study. We run Focus and our detector on the same Llama3-8B generations and entity annotations.
Beating Focus means a higher AUPRC overall. The larger, pre-registered gain should appear on the swap subset and on high-confidence errors, defined as entities in the bottom tertile of token entropy. We also report the uncertainty measures from the same study that are simpler to run and faithful to the originals: likelihood, entropy and claim-conditioned probability. HalluEntity reports these as weaker than Focus, but we do not have their exact numbers.
The sampling-consistency baseline resamples the full generation and checks whether each claim reappears. It is the family that DRAD compares against on WikiBio GPT-3; the file gives no numbers for it. It is the strongest conceptual rival, because our method also samples. We therefore give it exactly the same number of model calls as lookalike probing uses per target, with $k$ lookalikes times $m$ samples reallocated to resampling the target.
The retrieval-based detector is the DRAD entropy trigger together with BM25 retrieval. It serves as a reference point with privileged access to external text, not as an equal competitor.
For mitigation, the baseline is calibrated CSS on LongFact. It reports an overcommitment-aware utility of 0.913 against 0.846 without backoff, specificity retention of 0.938, and support precision of 0.9865 (Answer Only as Precisely as Justified). We rerun CSS with its support estimator and our generator. Beating it means higher utility at matched support precision, or equal utility with fewer swapped values surviving.
Two ablations isolate the key ingredient. Random-distractor probing replaces lookalikes with random entities of the same type. Discriminability-free scoring sets $\delta \equiv 1$. If random distractors perform as well as lookalikes, the gain comes from extra sampling, not from modeling confusable neighbors.
Why not just…?
Why not just resample the answer and check self-consistency? A swap comes from a stable association the model learned, so the wrong value tends to reappear in most samples. This is the "high-confidence hallucination" regime that DRAD itself concedes its entropy trigger misses. Self-consistency asks whether the model agrees with itself. Lookalike probing asks whether the answer is specific to this entity. The compute-matched comparison on the swap subset settles which question pays off. If consistency wins even there, our hypothesis is wrong.
Why not just retrieve the target's Wikipedia page? When retrieval is available and the page exists, it will usually win, and we report it as an upper reference. There are two reasons it is not the whole answer. Long-tail targets often lack good pages. Queries built only from local context can pull back the lookalike's page, which DRAD does not guard against. We will measure how often BM25's top hit is the lookalike on swap cases. A second experiment adds lookalike names to the query as negatives, which tests whether our signal also improves retrieval.
Why not just use contrastive decoding against a distractor prompt? Subtracting a distractor-conditioned distribution, in the spirit of Instruction Contrastive Decoding, would push down every token the target shares with its neighbors, including true shared facts such as field or nationality. It also doubles decoding cost at every step. The $\delta = 1$ ablation is effectively a post-hoc version of this naive contrast, and it will show how many true shared facts get flagged when discriminability is ignored.
Why not just ask the model which entity it might be confusing this with? This is a real threat, because it is cheaper. It will run as a single-prompt baseline: "Could this claim be about a similar entity instead? Answer yes or no." If it matches lookalike probing, the explicit belief comparison is unnecessary and we will say so.
Proposed Experiments
The first decisive check runs on ERBench, where lookalike sets and swap labels are exact. We will report what fraction of wrong answers are neighbor values, and the detector's AUPRC with model-generated versus database lookalikes. The gap between the two measures how much the method depends on the model knowing its own confusions.
The second check is HalluEntity with Llama3-8B against Focus's 0.40 AUPRC. Results will be broken down by swap versus non-swap errors and by entropy tertile. The prediction is modest gains overall and large gains on low-entropy swaps. Gains spread evenly across all error types would suggest we are measuring something other than confusion.
The third check is biography generation on FActScore entities, bucketed by rarity. This tests whether the swap share, and our advantage, grow toward the long tail, where FActScore shows precision is worst.
The fourth check is LongFact mitigation against calibrated CSS, reporting overcommitment-aware utility, specificity retention, support precision and model calls per response.
Model-generated lookalike lists may themselves be hallucinated. We therefore report how often listed lookalikes exist, and how detection degrades as that rate falls.
References
- HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detection
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models
- SLINT+ results for OAEI 2013 instance matching
- It's Not Polite to Point: Describing People with Uncertain Attributes
- Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems
- Mitigating Entity-Level Hallucination in Large Language Models
- Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding