Grounded Guessing: Detecting Cold-Entity Hallucinations via Retrieval Instability

Retrieval-augmented generation (RAG) — pairing a language model with a search system whose results are pasted into the prompt before answering — is the standard remedy for LLM hallucination, the phenomenon where a model states falsehoods fluently and confidently. The remedy works only when the document collection actually contains evidence about what was asked. This proposal studies the regime where it does not: questions about cold entities, things so new or so rare that no document describes them — a compound synthesized last month, a target with no literature footprint. Drug-discovery research has mapped this regime precisely for specialized predictors: Fokoue et al. and Nguyen et al. show that models trained on drug–target interactions collapse when asked about molecules or proteins unseen in training, and the systems built on language-model text embeddings, such as DrugLM and BAITSAO, openly concede that their signal vanishes for novel compounds with no textual record. For retrieval-augmented LLMs, this regime is unmeasured. Our hypothesis is that for a cold entity the retriever does not fail loudly; it fetches documents about similar entities, and the generator produces a confident, well-cited answer built on look-alikes — grounded guessing, which is harder to notice, and plausibly worse, than answering with no documents at all.

The second ingredient turns a known defect into an instrument. Prompt sensitivity — output instability under semantically meaningless changes in wording — is normally treated as noise to be suppressed: The Butterfly Effect of Altering Prompts shows a trailing space or a politeness token flips deterministic classifications, and When Benchmarks are Targets shows option reordering shifts leaderboard rankings by up to 8 positions. Crucially, Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity shows that retrievers themselves fetch divergent document sets under small query perturbations. We propose that this instability is not uniform across queries: it should concentrate on cold entities, where the retrieval neighborhood is a crowd of near-miss look-alikes rather than one clear target. Paraphrase-induced flip-flopping of the retrieved set can therefore serve as a cheap, model-agnostic detector of exactly the queries where grounding is illusory, and as the basis for abstention.

Core Claim

Grounding reduces hallucination for warm entities (those well covered by the corpus) but increases confident falsehood for cold entities, while retrieval-confidence scores remain high and uninformative — the retriever happily returns nearest neighbors that are similar-but-wrong. A hint of this already exists in uncontrolled form: on the ArchEHR-QA clinical benchmark, the DKITNLP pipeline improved answer relevance (30.50 versus 27.80 ungrounded) while slightly reducing factual accuracy (32.70 versus 33.60), despite correct evidence being in the prompt. No reported result found on Lacuna for hallucination rates stratified by an entity's footprint in the retrieval corpus, nor for any abstention mechanism conditioned on entity novelty; establishing both is the contribution.

The detection signal is simple. For a query $q$, generate $N$ paraphrases, retrieve the top-$k$ documents for each, and measure retrieval instability as

$$D(q) \;=\; 1 - \frac{2}{N(N-1)} \sum_{i<j} \mathrm{Jaccard}(R_i, R_j),$$

where $R_i$ is the top-$k$ document set for the $i$-th paraphrase and Jaccard is the overlap fraction between two sets. $D \approx 0$ means every paraphrase finds the same evidence (a warm entity with a clear retrieval target); $D$ near 1 means the paraphrases fish in different parts of the look-alike crowd. We abstain — or switch to an explicit "no direct evidence found" answering mode — when $D(q) > \tau$, with $\tau$ calibrated on warm development data to a target error rate. The claim is that $D$ separates cold-entity queries from warm ones far better than retrieval similarity scores do, and that gating on $D$ improves cold-slice factuality at matched answer rates.

Research Program

The first deliverable is a benchmark with controlled warm/cold splits, importing the split discipline from drug–target interaction work into LLM evaluation. ColdDTI distinguishes cold drug, cold protein, and cold pair settings; we port exactly these definitions to a question-answering corpus. The primary instantiation is drug-interaction QA, built from the DrugBank interaction triples used by DrugLM and the 600,000-plus drug–food interaction pairs behind ArcDFI: natural-language questions about interactions and mechanisms, with an entity made "cold" by removing all of its documents from the corpus and selecting entities that post-date the generator's training cutoff (verified by probing the closed-book model, so its parametric memory is also cold). A second instantiation reuses ArchEHR-QA directly, with a cold slice defined by evidence sparsity in the patient notes. A third uses Natural Questions with entity-frequency-defined coldness so results are comparable to TRAC.

The second deliverable is a measurement study: hallucination rate and answer-flip rate under paraphrase, as a function of coldness (fully cold, partially covered, warm), for both a closed-book generator and a standard two-stage RAG pipeline. This tests the grounded-guessing hypothesis directly — factuality should degrade with coldness for the grounded system while fluency and relevance stay flat, replicating the ArchEHR relevance/factuality divergence under controlled conditions. A companion analysis plots calibration curves of retrieval similarity scores against factual correctness, stratified by coldness; the hypothesis predicts the curves are flat-to-worse exactly where we need them most.

The third deliverable is the method: the paraphrase-divergence gate above, plus an intervention study comparing three behaviors on high-$D$ queries — abstain, answer with an explicit no-direct-evidence scaffold that forces citation of dissimilarity, or answer from the look-alike documents as usual. For interaction-claim questions we additionally test a hybrid that consults a structural cold-start predictor as a second opinion, since Song et al. show pre-trained protein models retain signal on completely cold targets.

Baselines

The naive reference point is the closed-book generator: the same LLM answering with no retrieved documents at all. On ArchEHR-QA, a zero-retrieval LLaMA 3.3 70B scored 30.7 overall (DKITNLP), so this is a real competitor, not a strawman; to be fair on compute, it gets the same $N$-pass budget spent on self-consistency sampling rather than paraphrase retrieval. Beating it means higher cold-slice factuality at an equal or higher answer rate.

The strongest existing system is the two-stage RAG pipeline of DKITNLP: a BioBERT bi-encoder retrieves candidate sentences, a fine-tuned cross-encoder re-ranks them (precision 51.45%, recall 89.86%, F1 65.44% on essential-sentence identification), and a Mistral-7B generator answers few-shot; it scored 31.6 overall. We run this pipeline verbatim on our corpora with the same generator, corpus, and tuning budget, and our method is this pipeline plus the divergence gate — so any difference is attributable to the gate alone. Beating it means higher cold-slice factuality with no warm-slice loss at matched coverage.

The cheapest serious competitor is retrieval-confidence thresholding: abstain whenever the top similarity score falls below a threshold, with the threshold searched on the same development data with the same compute we spend on paraphrases. The fourth is a statistical one: TRAC, a conformal-prediction wrapper for RAG that achieved empirical coverage of 0.90 at $\alpha = 0.10$ with mean prediction-set size 20.4 on Natural Questions, adapted to our single-answer selective setting. Beating both means better cold-slice conditional coverage at equal or lower abstention cost. Finally, two ablations isolate the key ingredient: a paraphrase ensemble with frozen retrieval (paraphrases shown only to the generator over one fixed document set), which separates retriever-side from generator-side sensitivity, and generator-only self-consistency with no retrieval signal at all. If the frozen-retrieval ablation matches the full method, the retriever flip-flop — our central claim — is not where the signal lives, and we would report that.

Why not just…?

Why not just use the specialized cold-start predictors? For pure interaction scoring, they win: ColdDTI reaches AUC 0.751 on cold-protein BindingDB and 0.818 on cold-pair Human, and ArcDFI reaches AUROC 0.9717 on cold-drug drug–food interactions. But these models emit a score, not a grounded natural-language claim with citations — the artifact clinicians and chemists actually consume — and SMDD-Bench shows frontier LLMs already collapse to near 0% on genuinely novel 3D molecular reasoning, so language-level claims are where LLM systems are deployed regardless. We include the structural models as oracle and verifier components, not competitors, and scope our claims to factual assertion quality, not binding-affinity accuracy.

Why not just threshold retrieval confidence? This is the real threat, and the calibration analysis exists precisely to settle it. The grounded-guessing hypothesis predicts high similarity scores for cold entities — nearest neighbors of a look-alike crowd are confidently similar — so the score should be uninformative exactly where it matters. If, in the experiment, similarity thresholding matches the divergence gate at equal compute on the cold slice, the gate should be dropped and the simpler method adopted; we treat that outcome as a legitimate result of the proposal.

Why not just improve retrieval? Query-expansion and hybrid-indexing pipelines such as Boros et al., and the cross-encoder re-ranking already in the DKITNLP system, improve precision when relevant documents exist. For a cold entity the target document does not exist, so better re-ranking only selects the best look-alike with more confidence — consistent with the observed factuality drop (32.70 grounded versus 33.60 ungrounded) despite good re-ranking. The warm-slice results will show whether improved retrieval erodes our gate's advantage where evidence genuinely exists; if it does, the gate's value is honestly confined to the cold regime.

Why not just use conformal prediction? TRAC already provides distribution-free coverage guarantees. But its guarantee is marginal — averaged over the query distribution — and its 0.90 coverage came with prediction sets of 20.4 answers, unusable as a single response. Nothing in the framework constrains the conditional error rate on the cold slice, which can be arbitrarily bad while average coverage holds. Our conditional-coverage experiment measures that gap directly for both methods.

Proposed Experiments and Decisive Checks

Experiment one plots factuality against coldness for closed-book, standard RAG, and gated RAG on the drug and clinical benchmarks. Decisive outcome: RAG factuality falls below closed-book on the cold slice while exceeding it on the warm slice, confirming grounded guessing. Experiment two reports the AUROC of $D$ for detecting cold-entity errors, alongside the AUROC of retrieval confidence and of generator self-consistency; no reported result found on Lacuna for any comparable detector, so this number is itself a contribution, and we pre-commit to reporting the confidence baseline honestly if it wins. Experiment three produces risk–coverage curves: factuality as a function of the fraction of questions answered, with the gate, thresholding, TRAC-adapted conformal, and no abstention, at matched $N$-pass budgets. Success is higher cold-slice factuality at matched answer rates with warm-slice factuality no worse than the ungated DKITNLP pipeline. Experiment four measures conditional coverage — achieved error rate on the cold slice at a nominal $1-\alpha$ guarantee — for conformal versus our gate, directly testing whether marginal guarantees hide the cold failure. Experiment five compares abstention, the no-direct-evidence scaffold, and look-alike answering on high-$D$ queries, deciding whether the right response to detected coldness is silence or an explicit, honest statement of insufficient evidence. Together these experiments either establish paraphrase-induced retrieval instability as a practical cold-entity hallucination signal, or cleanly hand the field the simpler method that beats it.