Scientific misinformation often works by quoting a paper correctly and then overstating it. A claim such as "drug X cures condition Y" can copy the exact wording of a paper's results section. The evidence that undercuts it, for example "the trial had 40 participants and no control arm," sits in the methods or limitations section and shares almost no words with the claim. Search systems readily find the matching passage, which the literature calls the "kernel of truth," and miss the passage that undercuts it. Once a language model is handed only the kernel, it tends to believe the false claim. We propose to score all passages of the source paper together in a single transformer pass. Each passage is marked with its own special token, and passages are allowed to attend to one another. The model can then reach the undercutting passage through its connection to the kernel passage rather than through the claim, which it barely resembles. The same pass returns a verdict on the claim.
The core bet is specific. In earlier work on packing many candidates into one input, attention between candidates turned out to be unnecessary. In fact-checking with asymmetric evidence, we expect that same attention to be the ingredient that closes the vocabulary gap. This matters because retrieval-augmented generation, meaning a language model answering with retrieved passages pasted into its prompt, is increasingly how people check claims. The known failure is severe. In Grounding Fallacies Misrepresenting Scientific Publications in Evidence (MISCIPLUS), Llama 3 labels misinformation TRUE 0.0% of the time from its own knowledge but 44.4% of the time once shown the retrieved kernel passage. Retrieval made the model worse.
Core Claim
We claim that the passage that undercuts a misleading claim is lexically far from the claim but close to the kernel passage. Both discuss the same measurement, cohort, or outcome. A model that sees the claim and every candidate passage in one sequence can therefore find the undercutting passage by following a two-step path, from claim to kernel and from kernel to limitation, inside its attention layers. Scoring each passage against the claim alone, whether by keyword search, dense embeddings, or independent cross-encoders, cannot follow that path.
The claim has a testable edge. Recall, Expand, and Multi-Candidate Cross-Encode (MCCE) packs thousands of entity-type candidates into one pass and reports that attention between candidates can be removed with no loss. Our hypothesis predicts the opposite result for evidence that undercuts a claim. If masking candidate-to-candidate attention does not hurt retrieval of the undercutting passage, the hypothesis is wrong.
Relation to Prior Work
The method comes from the joint-encoding line. Do It Once concatenates a dialogue context with all response options. It places a marker token before each option, reads each option's score from that marker's hidden state, and applies one softmax. It reports a 6-point absolute gain in Recall@1 over scoring options separately on MuTual, and it roughly halves training time. It also introduces permutation augmentation to counter order bias. MCCE extends the same idea to label spaces of more than 10,000 types. We reuse the marker-token readout and the permutation augmentation. We change what the candidates are: passages from one scientific paper rather than responses or type names. We also change what the model decides: a role for each passage plus a verdict on the claim, rather than a single choice.
The motivating evidence comes from two sources. MISCIPLUS supplies the phenomenon and the benchmark. Its retrievers find the kernel passage with mean reciprocal rank (MRR) up to 0.664 but fail on undercutting passages. The failure holds for BM25, which is keyword-overlap search, and for dense retrievers such as BioBERT and INSTRUCTOR. MISCIPLUS proposes no architecture to fix this, which is where we step in. WikiQA shows that the vocabulary gap is common and not only adversarial: 20.3% of correct answers share no content words with the question.
One result constrains the design and could contradict our proposal outright. HaluEval reports that showing a model a correct and a hallucinated answer side by side often lowers its detection accuracy compared with judging each alone. This contrast paradox cuts directly against joint encoding. Our response is that HaluEval's contrast was unstructured natural-language prompting of a model that was never trained to compare candidates. Do It Once gets its gain from trained marker tokens. Which explanation holds is an empirical question, and we test it directly rather than assume it.
Several pipelines reduce stages in neighbouring settings, and we differ from each. JointCQ generates claims and search queries in one pass, but retrieval and verification still run as separate stages. RAG-Zeval produces the whole faithfulness judgment in one structured output, but it assumes the relevant context has already been supplied. From RAG to Riches interleaves generation with exact-substring retrieval in a single decoding pass. About 40% of its errors are index failures caused by wording mismatch, which is exactly the gap we target. Multiview Identifiers Enhanced Generative Retrieval bridges the gap with synthetic pseudo-queries, but it retrieves only and scores candidates one at a time. None of these jointly encodes competing pieces of evidence so that one piece can locate another.
Research Program
Model. The model is a verifier with a long input. The sequence is the claim, then a [CLAIM] token, then every passage of the source paper. Each passage is prefixed by a [P] marker and its section heading. We will build it on two backbones: a long-context encoder, and a 7B decoder whose marker hidden states are read out as in Do It Once. For each passage, the marker's hidden state feeds a three-way classifier with the classes kernel, undercutting, or irrelevant. The [CLAIM] state feeds a verdict head with the classes supported, misleading, or unverifiable.
Handling length. Many papers exceed the context window. When they do, we first select a broad candidate set of passages using a cheap recall stage, as MCCE does. We report the recall ceiling of that stage separately so that its losses are not blamed on the joint model.
Training data. MISCIPLUS is small: 114 arguments over 100 papers. We therefore reserve it entirely for evaluation and build synthetic training data from open-access papers. An LLM writes an overgeneralized claim from a results passage. It then nominates which methods, limitation, or discussion passages undercut that claim. Those nominations are kept only if a second model agrees and the passage has low word overlap with the claim. We will hand-audit a sample of these labels and report label precision. We will also run permutation augmentation over passage order.
Analysis. For correctly retrieved undercutting passages, we will trace the attention flowing from their markers. The question is whether that attention routes through the kernel passage, as the hypothesis predicts, or directly to the claim.
Baselines
The naive reference point is standard retrieval followed by generation, run as in MISCIPLUS. BM25 and a dense retriever each return top-k passages for the claim, and a Llama 3 generator gives the verdict. The reported figures are kernel MRR up to 0.664 and a 44.4% misinformation acceptance rate for Llama 3 when given the kernel. The file gives no numeric MRR for undercutting passages, only that retrievers fail, so we will re-measure that figure ourselves. Beating this baseline means a clearly lower acceptance rate at matched passage budgets.
The strongest existing method for scoring passages is an independent cross-encoder. We will use the same backbone as our model, fine-tuned on the same synthetic data, scoring each (claim, passage) pair in its own pass. This is the direct counterpart of the independent-matching baseline in Do It Once, and it is what a careful engineer would build first. It gets the same training data, the same number of training steps, and the same hyperparameter-search budget as our model. Beating it means higher MRR and Recall@5 for undercutting passages on MISCIPLUS, and a lower false-acceptance rate.
A second strong competitor is a kernel-conditioned two-hop cross-encoder. It first predicts the kernel passage, then scores every other passage against the claim together with that kernel. We run it with a predicted kernel and, as an upper bound, with the gold kernel. This baseline implements our proposed path explicitly across two passes, so it is the decisive test of whether a single joint pass adds anything beyond efficiency.
A long-context LLM judge also serves as a baseline. We give GPT-4o the full paper and ask for the kernel passage, the undercutting passages, and a verdict, under the same passage segmentation. This tests whether a frontier model's untrained comparison suffers the contrast paradox that HaluEval reports.
The key ablation is our own model with candidate-to-candidate attention masked out, in the style of MCCE. Passages can then see the claim but not each other. The claim predicts a large drop in retrieval of undercutting passages and little change in kernel retrieval.
Why not just…?
Why not just use the kernel passage as a second query? This is the real threat. It implements our hypothesized path explicitly, and with a gold kernel it may match us. We treat it as the decisive comparison and do not argue around it. If the predicted-kernel two-hop system matches the joint model, our contribution shrinks to latency and a single-pass verdict, and we will say so. We expect the joint model to win when the kernel prediction is wrong or split across passages. In those cases the two-hop system passes its error forward, while the joint model can weigh several partial kernels at once. The gold-kernel versus predicted-kernel gap will show how much of this is at stake.
Why not just paste the whole paper into a frontier LLM? HaluEval shows that untrained side-by-side comparison can lower detection accuracy. ChatGPT reached only 62.59% on QA hallucination detection even without side-by-side contrast. MISCIPLUS further shows that kernel passages actively mislead models. The long-context judge baseline measures whether this still holds when the model has the whole paper. If GPT-4o matches us, the case for a trained 7B verifier rests on cost and consistency. Prompt engineering in consistency and reliability shows the relevant instability: Fleiss kappa, a measure of agreement across repeated runs, ranged from -0.002 to 0.984 across prompts. We will report agreement across repeated runs for the judge alongside its accuracy.
Why not just train a better retriever or query rewriter? RaFe trains rewriters to produce keyword-rich queries that match what a reranker prefers. That pushes queries further toward the claim's own vocabulary, which is the wrong direction when the evidence is phrased differently from the claim. MISCIPLUS already shows that dense retrievers fail on undercutting passages. We will include a RaFe-style rewriter in front of the independent cross-encoder to confirm that rewriting alone does not close the gap.
Proposed Experiments
Main test on MISCIPLUS. We measure MRR and Recall@5 separately for kernel and undercutting passages, together with the false-acceptance rate on misinformation claims. We also count forward passes per claim. We expect gains to concentrate in undercutting retrieval. The primary decisive check is the attention-masking ablation. The secondary decisive check is the comparison against the predicted-kernel two-hop system.
Direct test of the contrast paradox. On HaluEval QA, the same backbone will judge a correct answer and a hallucinated answer in three ways: in isolation, jointly with trained marker tokens, and jointly through a plain natural-language prompt. The reference point is ChatGPT's 62.59% QA detection accuracy. If trained markers reverse the paradox while the plain joint prompt reproduces it, that settles the tension between HaluEval and Do It Once.
Sanity check that the gain is about the vocabulary gap. On WikiQA sentence selection, we will measure the joint model's MAP on the zero-overlap subset versus the rest. The reference is CNN-Cnt's reported MAP of 0.6520 on the full set. The hypothesis predicts that the joint model's advantage over independent scoring is largest on the zero-overlap subset.
Transfer and length stress tests. We will examine how far synthetic training transfers by evaluating on a held-out scientific domain. We will also examine how performance degrades as papers exceed the context window and the first-stage recall of passages falls.
References
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence
- Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing
- Do It Once: An Embarrassingly Simple Joint Matching Approach to Response Selection
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- JointCQ: Improving Factual Hallucination Detection with Joint Claim and Query Generation
- RAG-Zeval: Enhancing RAG Responses Evaluator through End-to-End Reasoning and Ranking-Based Reinforcement Learning
- From RAG to Riches: Retrieval Interlaced with Sequence Generation
- Multiview Identifiers Enhanced Generative Retrieval
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
- RaFe: Ranking Feedback Improves Query Rewriting for RAG