Chain-Closed Evidence Extraction for Retrieval-Augmented Reading of Scientific Text

Language models that answer questions from retrieved documents fail in a specific and underexplored way on scientific text. The sentence that holds the answer often refers to its subject only indirectly, as in "it was annealed at 900 °C", "the other two samples", or "this oncogene". The antecedent, meaning the earlier mention that tells the reader which entity is meant, sits several sentences or paragraphs away. Retrieval also places passages from other papers in the same prompt, and those papers mention their own "alloy", "protein" or "sample". The model then picks up a fact from the correct sentence and attaches it to the wrong entity. Current noise filters select relevant sentences without checking whether those sentences can still be understood on their own. Current coreference tools rewrite the text before the model sees it, and any mistake they make becomes permanent.

We propose training a single reader model to do two things at once. It extracts the evidence sentences, and for every indirect mention in that evidence it points to the sentence containing the mention's antecedent. Training enforces chain closure: the extracted evidence is only complete when every indirect mention in it can be traced to a named entity in the same source document. The training distractors are chosen so that they collide with the target entities, meaning they come from other papers and share entity types and surface names with the target document. The central claim is that, in technical domains, much of the damage from distracting context is really coreference damage. It should therefore be fixed by structural supervision on entity chains, not by more filtering or more context.

Core Claim and Where It Sits

Three lines of work each cover part of this problem, and none covers all of it. Aligning Extraction and Generation for Robust Retrieval-Augmented Generation (Ext2Gen) trains a model with preference optimization to extract salient sentences from up to 25 distractor chunks before answering. It is our method's direct ancestor and its strongest competitor. However, it treats sentences independently, so an extracted sentence that depends on an unextracted antecedent stays ambiguous. Bridging Context Gaps (LQCA) resolves coreference with an off-the-shelf resolver and replaces pronouns in the text. It shows that this helps general long-context QA, with an average gain of +3.61% on GPT-4o and +6.1% on mid-document evidence. It also documents the cost: upstream errors are written permanently into the text, and the method was never tested on specialized nomenclature or on distractors from external documents. CLAP applies coreference resolution at indexing time to repair retrieval. It does nothing for the reader once noisy passages arrive, and it inflates the index 20 to 32 times.

On the domain side, two papers supply the motivation. CRAFT coreference annotation shows that following identity chains in biomedical articles adds 76% more extractable entities and that general-domain resolvers perform poorly there. LitXBench shows that frontier models extracting materials experiments still fail on indirect references such as "the other two samples".

BioCoref constrains the design in two ways. First, adding unstructured introductory context hurt biomedical coreference while structured entity dictionaries helped, so our method must not simply add more context. Second, more than 76% of coreference relations fall within 500 words. That makes a cheap fixed context window a serious alternative, which we test directly below.

Our method differs from Ext2Gen in what it extracts: a closed evidence graph, not a sentence set. It differs from LQCA in where coreference happens and whether it is reversible: linking is done by the reader, and the links are pointers the answer generator can override, not rewrites of the text. It differs from all three in its training signal, which is distractors built to cause entity collisions in technical domains.

Research Program

The reader receives a query and a set of retrieved passages. Each sentence is tagged with a document identifier and an index. The reader emits three things before answering: a list of evidence sentence identifiers, a set of link pairs mapping each indirect mention in the evidence to an antecedent sentence identifier in the same document, and then the answer.

Evidence is chain-closed when following the links from every indirect mention ends at a named or canonical mention within the extracted set. Supervision for the links comes from gold coreference chains. In biomedicine these are CRAFT's identity chains. In materials science we derive them from LitXBench's structured sample lineage graphs, in which each extracted measurement is tied to a specific sample object in the paper. Outputs whose evidence is not closed, or whose links cross document boundaries, are used as the rejected side of preference pairs. The chosen side is the closed extraction followed by the gold answer. Training uses Direct Preference Optimization (DPO), the same algorithm Ext2Gen uses, so the only differences from Ext2Gen are the output structure and the data.

Distractors are built in three tiers. The first is random chunks from other papers, matching Ext2Gen's setting. The second is chunks matched by topic. The third is entity-collision chunks: passages from other CRAFT articles or other alloy papers that contain mentions of the same entity type and overlapping surface forms. These are the cases where a sentence-level relevance filter has no basis for preferring one "the protein" over another.

Questions are generated from held-out gold chains. Each question is answered by a sentence that refers to its entity only indirectly, and the antecedent sits at a controlled distance. This yields a test set stratified by antecedent distance (within 500 words or beyond) and by distractor tier.

The base model is Llama-3.1-8B-Instruct. We choose it because Ext2Gen reports on it and because LDAR found 8B models much more fragile to noise than GPT-4o, so this is the regime where the effect should be largest. We also run GPT-4o-class models with prompting only, to check whether the structured output format helps without any training.

Baselines

The naive reference is the untuned Llama-3.1-8B-Instruct reading all retrieved passages, with and without a system prompt telling it to ignore irrelevant content. Breaking Focus reports that such instructions do not prevent an average degradation of about 45% under coherent distractors, so we expect the prompt to change little. The run establishes the size of the collision penalty on our data.

The strongest existing method is Ext2Gen-style joint extraction and generation. On its own evaluation sets, the cited paper reports an average score of 0.557 for Ext2Gen-R2 against 0.473 for default Llama3.1-8b-instruct. We first reproduce that gap on Ext2Gen's PubMed subset as a sanity check. We then retrain Ext2Gen on exactly our training questions, including all three distractor tiers, with the same DPO steps, learning-rate sweep and preference-pair count as our model. This removes any advantage that would come only from our collision data. Beating it means a statistically significant gain in answer accuracy on the collision tier and on antecedents beyond 500 words, without losing more than one point on Ext2Gen's own PubMed set.

The pipeline competitor is LQCA-style rewriting followed by the same Ext2Gen reader. We run it twice: once with LQCA's general resolver, and once with a resolver given BioCoref's best structured prompting strategy, entity-aware prompting. For that strategy BioCoref reports 0.740 F1 with LLaMA-17B, and 0.791 F1 for LLaMA-8B in the local-only setting. No reported result was found on Lacuna for LQCA on any biomedical or materials corpus.

Three ablations isolate the key ingredient. The first, closure supervision without collision distractors, tests whether the data or the structure does the work. The second, collision distractors without closure supervision, is the retrained Ext2Gen already described. The third, a fixed-window extractor, appends the preceding 500 words of the source document to every extracted sentence and then generates, with the same training. An oracle that supplies gold chains to the reader gives the upper bound.

Why not just…?

Why not just resolve coreference first and feed the rewritten text to a noise-robust reader? This composition of LQCA and Ext2Gen is the obvious approach, and it is a real threat when the resolver is accurate. Its weakness is that a wrong replacement cannot be undone, and BioCoref and CRAFT both show that general resolvers degrade badly on biomedical text (SpanBERT-large reaches 0.1322 F1 on CRAFT in BioCoref). We will measure answer accuracy as a function of resolver error, using gold chains corrupted at controlled rates. Our claim requires the rewrite pipeline to lose accuracy faster than our pointer-based reader as corruption rises. If it does not, the rewrite pipeline is the better engineering choice and we will say so.

Why not just pull in a fixed window of preceding text around each extracted sentence? Since BioCoref finds over 76% of relations within 500 words, this cheap heuristic may recover most antecedents. It is the alternative most likely to match us on short-range cases. Against it stand BioCoref's own context paradox, where extra unstructured context reduced precision, and the expectation that windows re-import distractor-like material. The distance-stratified test settles the question. If the window matches our method within 500 words and also beyond it, chain closure adds nothing, and we would narrow our claim to the long tail or drop it.

Why not just prepend a domain entity dictionary? BioCoref shows that structured dictionaries help. A dictionary can say that "the protein" is likely a known gene product, but it cannot say which paper's protein is meant when collision distractors share the same vocabulary. We include dictionary-augmented prompting and dictionary-augmented Ext2Gen as baselines. Our prediction is that dictionaries close part of the gap on random distractors and little of it on the collision tier.

Why not just train on more distractors with DPO, as Breaking Focus does? This is exactly the "collision distractors without closure" ablation. If it matches the full method, the gain comes from the data, not the structure, and that would be the honest finding.

Proposed Experiments and Decisive Checks

The primary test set is built from CRAFT, using held-out articles only, together with LitXAlloy papers into which we inject retrieved passages from other alloy papers. The PubMed portion of Ext2Gen is used to check for regression.

We report several metrics. Answer accuracy is reported per distractor tier and per antecedent-distance bin. Collision attribution rate is the fraction of answers whose fact is correct but whose entity belongs to a distractor document. Closure rate and link precision are reported against gold chains. For LitXAlloy we report measurement F1 under Hungarian matching. LitXBench reports 0.80 F1 for Gemini 3.1 Pro on clean, closed-context extraction, which bounds what a noisy, retrieval-based setting can be expected to reach. We also report tokens consumed relative to full context, to compare cost against LDAR's reported use of 25% to 63% of full-context tokens.

Three outcomes would decide the matter. The first is whether the full method reduces collision attribution relative to retrained Ext2Gen by more than it reduces ordinary relevance errors; that pattern would show coreference is the operative mechanism. The second is whether the advantage over the fixed-window extractor concentrates beyond 500 words. The third is whether accuracy degrades more slowly than the rewrite pipeline under injected link errors. If all three hold, entity-chain supervision is a distinct remedy for distraction in technical text. If the window baseline or the data-only ablation matches the full method, the cheaper fix wins and the paper reports that.