Reading or Remembering? Auditing Document-Level Relation Extraction with Outdated Facts

Document-level relation extraction means reading a multi-sentence passage and outputting facts such as (Cumberland, contains, magnetite), even when the two entities never share a sentence. Its main benchmark, DocRED, was built from 2019 Wikipedia articles aligned with Wikidata. Every modern language model has almost certainly trained on both the articles and the facts behind them. A high score could therefore mean the model linked evidence across sentences, or it could mean the model recognized the entity names and recalled a fact it already knew. Current evaluations cannot tell these apart, and the standard contamination tests were designed for isolated questions, not for facts spread across a document.

We propose to separate reading from recall using a signal that time has already created. Some DocRED facts have changed in the world since 2019. For those facts, and for controlled edits of the same documents, the facts diverge into three answers: the label the model may have memorized from the benchmark, the current real-world value the model may know from newer training data, and the value the text actually states. An extractor that reads should follow the text. One that recalls will return one of the other two answers, and which one it returns tells us whether the leak comes from benchmark memorization or from general world knowledge. The main hypothesis is that recall is concentrated in cross-sentence facts, which are exactly the cases where reading is hardest. If so, the reported progress on inter-sentence reasoning is partly inflated.

Core Claim

We claim that a three-way attribution test, applied to type-preserving entity swaps and to naturally updated revisions of the same documents, can measure contamination in cross-sentence extraction fact by fact, and that it gives answers neither of the existing signals can. Output-distribution collapse (Generalization or Memorization) flags that a model has memorized something, but not which extracted facts were read. Post-cutoff performance decay is fragile: Test of Time showed it disappears once questions are reformatted, even though influence-function analysis on OLMo2 confirmed the contamination was still there. Multi-sentence extraction is itself a reformatting of Wikidata triples into prose, so there is good reason to expect the same laundering effect here. That is why we avoid decay-based signals entirely.

The closest prior method is the trigger-substitution probe of Extracting or Guessing?. It swaps event words for synonyms in sentence-level temporal relation benchmarks (MATRES-DS) and subtracts trigger-only predictions to reduce reliance on memorized event orderings. We move this idea to multi-entity documents, substitute entities rather than triggers, and use real-world fact change as a third reference answer. The knowledge-obsolescence literature supplies the behavioral prediction. Can Agent Memory Systems Track Evolving State? found that models anchor on salient but superseded facts even when the context contains the correct update, and that chain-of-thought reasoning sometimes made this worse. SUMIE found that models updating entity tables from conflicting paragraphs produced 25% unsupported values. Neither study examined relation extraction across sentences, and neither separated benchmark leakage from world knowledge.

Research Program

Stage one: build the audit set from DocRED's human-annotated documents.

The first probe is the counterfactual swap. For a gold triple, we replace the object entity with another entity of the same type throughout every mention in the document. Gold labels are rewritten by the same mapping, so the chain of reasoning and any bridge entities stay intact while the correct answer changes. DocRED's evidence-sentence annotations tell us whether each fact is intra-sentence or inter-sentence, and which reasoning category it belongs to (bridge entity, coreference, or commonsense). This lets us break results down along exactly the axis the benchmark was built to test: DocRED reports that 40.7% of its facts require inter-sentence reasoning.

The second probe is natural obsolescence. We compare each DocRED triple against current Wikidata to find relations whose object has since changed, such as employer, team membership, or officeholder. For those articles, we align the current Wikipedia revision so that the updated fact appears in real, human-written prose rather than in our edits. Humans verify that the evidence spans in the revision actually state the new value. We do not know in advance how many DocRED triples have changed, and counting them is the first deliverable.

For every prediction, the attribution step assigns one of four outcomes: the text value (reading), the 2019 label (benchmark or 2019-era recall), the current Wikidata value (world-knowledge recall), or something else.

Stage two: validate the attribution against known training data.

On OLMo2-7B-Instruct, whose pretraining corpus is public, we measure how often each DocRED article and triple appears in that corpus. We then test whether the recall rate increases with exposure frequency. On a small sample of about 40 documents, matching the scale used in Test of Time, we run influence-function attribution to check that recall-labeled predictions trace back to training documents. This step is what licenses using the cheap, black-box audit on closed models.

Stage three: test one remedy.

We apply evidence-contrast decoding, following the counterfactual subtraction in Extracting or Guessing?. The model scores a candidate relation on the full document, scores it again on the document with its annotated evidence sentences removed, and subtracts the second score from the first. Relations the model would predict without evidence are penalized. For fine-tuned extractors, the remedy is simpler: we add swapped documents to training.

Baselines

The naive reference point is zero-shot prompting of an instruction-tuned model on the original DocRED documents. This is the setting where contamination should inflate scores the most, and it defines the gap the audit must explain.

The strongest task-specific baseline is ATLOP (Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling), which reports 63.4 F1 on DocRED. ATLOP learns a per-entity-pair decision threshold and pools only the context tokens that both entities attend to. It is the strongest competitor on this question because its pooling is explicitly designed to ground predictions in the document. If any system reads rather than recalls, it should be this one. We compare it with MRN (MRN), which reports 61.74 F1 with BERT, and with the PRiSM calibration codebase (brightjade/PRiSM) run on Re-DocRED, which corrects DocRED's missing labels (Revisiting DocRED).

For LLM-based extraction, we follow the prompting setups of AutoRE and LLM with Relation Classifier. We found no reported result on Lacuna for these methods' DocRED F1 in the file, so we will report our own reproductions.

All systems are fine-tuned or prompted on the same training split, with the same encoder size where applicable and the same hyperparameter search budget. All are evaluated on identical original, swapped, and updated documents. For reference, the original DocRED paper reports human F1 of 88.0 and a best baseline of 54.1.

We add one ablation to isolate the key ingredient, the third reference answer. It collapses the audit to a binary "follows the text or not" judgment. This shows how much diagnostic information the distinction between benchmark recall and world-knowledge recall adds.

A remedy counts as beating a baseline if it raises the text-following rate on swapped and updated inter-sentence facts by a margin larger than the variation across random seeds, and does so while staying within that same seed-variation margin of the baseline's F1 on original documents.

Why not just…?

Why not just evaluate on newly published documents after the training cutoff? New documents change entity popularity, domain, and annotation conditions all at once, so any drop in performance cannot be cleanly attributed to contamination. Test of Time also showed that post-cutoff decay vanishes when content is reformatted, even when contamination is mechanistically confirmed. Our design holds the documents fixed and changes only the answer, and the natural-revision set supplies the realism a temporal split would offer.

Why not just anonymize entities with placeholders? This is a real threat, because anonymization also blocks recall. Its cost is that it removes type cues and the commonsense knowledge a legitimate reader is expected to use, which DocRED counts as a reasoning category in its own right. As a result, it measures a harder, different task. We run anonymization as a comparison condition. If it yields the same inter-versus-intra-sentence gap as the entity swap at similar F1, we will say the simpler probe suffices for scoring. It would still not distinguish benchmark recall from world-knowledge recall, and it would not test behavior when the text conflicts with what the model believes, which is the behavior obsolescence produces.

Why not just run output-distribution contamination detection? Detection via output distribution (Generalization or Memorization) improved accuracy and AUC by 21.8% to 30.2% over n-gram baselines. However, it produces a verdict per prompt, not per fact, so it cannot say which of the relations in a document were read. We will test whether its per-document scores correlate with our recall rates. A strong correlation would validate both methods.

Why not just delete the evidence sentences? Deletion shows whether a model can recall a fact, but not whether it overrides the text when the two conflict. The failure that matters under obsolescence is the second one, as the state-drift findings show. Deletion is instead built into our remedy and reported as its own probe.

Proposed Experiments

The primary measurement is the four-way attribution rate. We report it by model family, by intra-sentence versus inter-sentence fact, by reasoning type, and by whether chain-of-thought prompting is used. The central test is whether text-following drops for inter-sentence facts relative to intra-sentence facts on swapped documents, while F1 on the original documents shows no such gap.

A second test compares naturally updated revisions with arbitrary swaps. Recall errors that land on the current Wikidata value point to world knowledge. Errors that land on the 2019 label point to benchmark or era-specific memorization.

A third test checks the salience-trap prediction from Can Agent Memory Systems Track Evolving State?: chain-of-thought should lower text-following on conflicting facts.

The decisive checks run in both directions. If text-following stays near ceiling for every system and fact type, document-level extraction scores are largely trustworthy, and we will publish that negative result. If recall rates on OLMo2 do not rise with corpus exposure, the attribution is measuring edit implausibility rather than memory. To guard against implausible edits, swaps are restricted to type-consistent entities and humans rate a sample for fluency. Finally, we will report a contamination-adjusted F1, computed only on facts that pass the swap test, alongside standard F1. This gives future work on cross-sentence extraction a number that reflects reading.