Event coreference resolution asks whether two pieces of text describe the same real-world happening. For example, "the plane went down near Lockerbie" in one news story and "the 1988 bombing" in another refer to the same event. The standard benchmarks for this task are built from old news (ECB+) and from Wikipedia (WEC-Eng). Both are text that today's pretrained models have very likely read, often many times. A model that recognizes the specific historical event behind each mention could link the mentions without comparing what the two passages say about who, where, and when. Benchmark scores would then partly measure recall of events the model has memorized, not resolution. The task is meant to be deployed on fresh news, where no such memory exists, so that portion of the score would not carry over.
We propose a controlled audit that separates these two sources of skill. The design crosses two factors. The first is whether the events happened before or after a model's training cutoff. The second is whether the named arguments in every mention (people, places, organizations, dates) are kept or are consistently replaced with fictional ones across the whole corpus. Renaming leaves all textual evidence for coreference intact but removes the cue that triggers memory. Renaming may also make the data harder for unrelated reasons. A pure distribution shift of that kind should hurt pre-cutoff and post-cutoff events about equally. Recall-driven skill should collapse only on pre-cutoff events. The gap between those two drops is our measure of how much benchmark performance comes from remembered events.
Core Claim
We claim that a measurable share of cross-document event coreference performance comes from recognizing the real-world event rather than comparing the passages. We expect this share to be large on WEC-Eng, whose mentions are often short references to famous events lifted from Wikipedia. We expect it to be smaller on ECB+, whose mentions sit in longer descriptive news text. The estimate we propose is a difference-in-differences:
$$\Delta_{\text{recall}} = \big(F_{\text{pre}}^{\text{orig}} - F_{\text{pre}}^{\text{ren}}\big) - \big(F_{\text{post}}^{\text{orig}} - F_{\text{post}}^{\text{ren}}\big)$$
Here $F$ is CoNLL F1, the average of three standard clustering-agreement scores. "pre" and "post" index events dated before or after the evaluated model's training cutoff, and "orig" and "ren" index original versus consistently renamed text. A positive $\Delta_{\text{recall}}$ means renaming hurts specifically where the model could have remembered the event. We also predict that $\Delta_{\text{recall}}$ grows with how strongly a model shows signs of having seen the mention's source passage, as measured by a membership-inference score.
This is not the same as asking "was ECB+ in the training data?" That question has an obvious yes-ish answer and says nothing about whether the exposure changes coreference decisions. Our question is whether the decisions depend on that exposure.
Prior Work and the Gap
Two strands motivate the gap. On the coreference side, Revisiting the Evaluation for Cross Document Event Coreference showed that ECB+ scores are inflated by evaluation conventions. Its fixes were to remove the gold sub-topic groupings and include singletons, meaning events mentioned only once. Under its SIMPLE-CDEC setting, a baseline fell from 69.2 to 42.3 CoNLL F1. That work exposed leakage internal to the benchmark, but it predates pretrained models and could not ask about leakage from pretraining. WEC: Deriving a Large-scale Cross-document Event Coreference dataset from Wikipedia built WEC-Eng automatically by treating Wikipedia hyperlinks to event articles as coreferent mentions. This is exactly the material most heavily represented in pretraining, and the paper did not test for that. What happens before and after: Multi-Event Commonsense in Event Coreference Resolution used GPT-3 to generate what happens before and after each mentioned event and reported gains on ECB+. Its authors did not check whether these "commonsense" inferences are recalled news narratives. Its observed failure, where two different actors performing the same action receive nearly identical inferences, suggests memory and argument structure are entangled.
On the contamination side, several papers supply our tools. Min-K%++ scores whether a passage was in pretraining by normalizing each token's log-probability against the vocabulary distribution. Proving Test Set Contamination in Black-Box Language Models detects benchmark exposure by testing whether a model prefers a dataset's published ordering over shuffled orderings. The WikiMIA construction used in Min-K%++ splits Wikipedia event articles by date relative to a model's cutoff, and this is the idea we reuse to build post-cutoff coreference data. One paper constrains our design. Back to Square One showed that entity-coreference models on the Winograd Schema exceed 60% accuracy even with the candidate entities removed. Coreference scores can therefore rest on cues unrelated to the reasoning the benchmark intends, so our controls must hold the textual evidence fixed rather than merely perturb it. DyVal regenerates reasoning problems procedurally to defeat memorization, but it explicitly stays in symbolic domains. Our renaming procedure is a modest, linguistically safe version of that idea for discourse.
Research Program
The first artifact is a post-cutoff WEC-style corpus. WEC's extraction pipeline is automatic: it selects event articles by infobox category, collects anchor texts linking to them, filters out dates and locations, and caps repeated strings. We rerun it on a recent Wikipedia snapshot, restricted to event articles created after a chosen cutoff. We then draw a date-matched pre-cutoff sample from the same infobox categories. Recent events have fewer inbound links and smaller clusters, so we match pre- and post-cutoff samples on cluster size, mention length, and category. As WEC did, we manually validate the test portions; that paper reported about 17% noise in its automatic training data. For ECB+, whose events are all old, we construct a smaller post-cutoff analogue only for the within-topic setting: a handful of recent news topics annotated by two annotators. We report this as a secondary result.
The second artifact is a consistent renaming transform. We tag named people, organizations, locations, and dates with off-the-shelf tools. Each surface entity maps to a type-appropriate fictional replacement that is constant across the entire corpus. Dates shift by a constant offset per cluster, so temporal relations between mentions are preserved. Event names that are themselves names, such as "the Lockerbie bombing," are rewritten through the same mapping. Every piece of textual evidence that two mentions share therefore remains shared. Two checks verify this. First, a head-lemma coreference baseline, which cannot use world knowledge, must score the same on original and renamed text. Second, annotators confirm on a sample that gold coreference judgments are recoverable from the renamed text alone.
The analysis runs the 2×2 design on two kinds of models. The first is a fine-tuned RoBERTa cross-encoder in the style of PAIRWISERL from Pairwise Representation Learning for Event Coreference. This model reads both mention sentences jointly and scores argument compatibility from semantic-role spans. The second is an open-weights decoder language model used as a pairwise judge, both prompted and fine-tuned. We prefer models whose pretraining data and cutoff are documented, such as the Pythia family on the Pile, because exposure can then be verified by lookup rather than only inferred. For each mention context we compute a Min-K%++ score and regress the per-pair renaming penalty on it.
As a side audit, we apply the sharded ordering test to the released ECB+ and WEC-Eng files. This tells readers whether the benchmark files themselves, not only the underlying news and Wikipedia text, were seen.
Baselines
The naive reference point is the head-lemma baseline, which links two mentions when their head words share a lemma, together with the Lemma-δ variant that adds document TF-IDF similarity, as defined in Revisiting the Evaluation for Cross Document Event Coreference. Neither has access to world knowledge, so their $\Delta_{\text{recall}}$ must be near zero, and any nonzero value is a measurement error in our renaming. We found no reported Lemma-δ figure on WEC-Eng or on the standard ECB+ split on Lacuna, so we report our own runs. Neural models should beat the lemma baseline on post-cutoff renamed text. If they do not, their apparent advantage on ECB+ and WEC-Eng is largely attributable to memory.
The strongest existing coreference systems are the RoBERTa pairwise scorer from WEC, which reports 62.3 CoNLL F1 on WEC-Eng and 80.8 on ECB+, and PAIRWISERL, which reports 84.4 CoNLL F1 on ECB+ and 57.12 average F on KBP. We retrain both from their published recipes on the original training splits, with the same encoder size and tuning budget. We first reproduce the reported numbers on the original test sets within about one point before running the 2×2. The comparison is not about beating them. It asks how much of their score survives the condition where recall is impossible: post-cutoff and renamed. We will regard memory as a minor factor if $\Delta_{\text{recall}}$ is within run-to-run noise, estimated from five seeds and bootstrap over clusters.
The key ablation isolates argument structure. We run PAIRWISERL with and without its semantic-role argument module. That paper reports the module adds only 0.4 F1 on ECB+. Our hypothesis is that this small gain is small partly because memory already supplies the argument information. If so, the argument module should matter much more on renamed pre-cutoff text than on original text.
Why not just…?
Why not just run Min-K%++ or the ordering test on the benchmark text and report contamination? Those tests answer whether the text was seen. For Wikipedia and decade-old news the answer is close to certain, and in any case it does not tell us whether exposure changes coreference decisions. Min-K%++ also has a documented weakness on passages seen only once, which describes many individual news articles. We keep both tests as covariates and as a side audit. The causal estimate, however, comes from the 2×2 design, which measures the effect on decisions directly.
Why not just paraphrase the test set with a language model? Paraphrasing keeps the named arguments, and those are precisely the cues that call up a remembered event, so it leaves the suspected mechanism untouched. It also risks altering semantic roles or turning identity into a looser associative relation. The EvoEval refactoring results surveyed in Benchmark Data Contamination of Large Language Models: A Survey, an average drop of 39.4% on HumanEval, show that rewriting exposes memorization in code. Such drops cannot be separated from added difficulty, however, because no unexposed control set is rewritten alongside. Our design supplies that control.
Why not just evaluate on post-cutoff events and skip the renaming? This is the most serious alternative, and it is half of our design. On its own it confounds exposure with everything else that differs about recent events: cluster sizes, link density, topic mix, and annotation noise from fresh extraction. A drop on post-cutoff data could reflect any of these. Renaming applied to both time periods cancels these differences. If renaming turns out to hurt pre- and post-cutoff data equally, then the time split alone was mostly measuring distribution shift, and we will say so.
Why not just evaluate under SIMPLE-CDEC? Removing gold topic boundaries addresses a different inflation, namely protection from same-word confusions across topics. It does not touch memory. We run every 2×2 cell under both the standard and SIMPLE-CDEC protocols, which lets us test whether the two inflations add up or overlap.
Proposed Experiments and Decisive Checks
The main experiment estimates $\Delta_{\text{recall}}$ for each model on matched WEC-style pre- and post-cutoff test sets, and secondarily on ECB+ with its small post-cutoff analogue. We report CoNLL F1 together with its three component scores, with and without singletons. Confidence intervals come from cluster-level bootstrap. Two measurement checks must pass. The lemma baseline must show near-zero renaming penalty in both periods, and the annotator sample must confirm that gold labels are recoverable from renamed text. If either fails, the renaming is breaking textual evidence and we revise it before interpreting anything.
The decisive result is the interaction term. A clearly positive $\Delta_{\text{recall}}$ for neural models, a stronger effect on WEC-Eng than on ECB+, and a positive slope of renaming penalty against Min-K%++ score would together show that the benchmarks partly reward event recall. A null interaction would be equally informative: it would let the community keep using these benchmarks with more confidence.
A final stress test creates conflict between memory and text. Each mention keeps the famous trigger phrase of a real event, but its arguments are swapped to those of a different real event from the same infobox category, so the text indicates non-coreference while memory indicates coreference. Falsely linking such pairs at a higher rate than renamed controls would be direct evidence that memory overrides textual comparison. We also apply this conflict test to the before/after inference generator from What happens before and after, which tells us whether its "commonsense" bridges follow the passage or the remembered story.
References
- Revisiting the Evaluation for Cross Document Event Coreference
- WEC: Deriving a Large-scale Cross-document Event Coreference dataset from Wikipedia
- What happens before and after: Multi-Event Commonsense in Event Coreference Resolution
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- Proving Test Set Contamination in Black-Box Language Models
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Pairwise Representation Learning for Event Coreference
- Benchmark Data Contamination of Large Language Models: A Survey