When a language model correctly answers "what happened first?" about a news story or an everyday script, we cannot tell whether it read the text or remembered the answer. Benchmarks for event and causal reasoning are built from old, widely shared text, and that text has very likely ended up in pretraining data. This proposal tests memorization by contradicting it. We take a familiar narrative and its annotated timeline, then edit the timeline in a controlled way: we reverse one "A happened before B" link, cut one cause, or add a decision branch. A model rewrites the story minimally to match the edited timeline, and the new version is told in shuffled order, with flashbacks and interleaved threads. We then ask the model under test about the events. On questions whose correct answer changed because of the edit, a model that answers from memory will give the original story's answer. A model that reads the text will give the new one.
This turns contamination from something we can only suspect into a behavior we can count. It also gives a direct test of an open dispute. Fresh post-cutoff news makes causal reasoning collapse, which suggests models are "causal parrots" (Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?). Yet multi-step synthesized questions show no post-cutoff decay at all (Beyond Memorization). If deeper questions remove reliance on memory, deep questions about rewired stories should show no pull toward the original answer. If they only hide that reliance, the pull should remain.
Core Claim
A directed temporal graph is a set of events connected by arrows meaning "happens before" or "causes." When a story has such a graph, editing the graph and re-rendering the story produces paired items. The two versions share almost all surface text but differ in known edges. On these pairs, the rate at which a model answers according to the original graph is a clean, item-level measure of how much it relies on memorized event knowledge. Three things make the measure interpretable. Each edit must be checked against the whole timeline, so the edited timeline contains no contradictions. Each rendering must be verified by converting the text back into a graph. And the same edits applied to stories the model cannot have seen must give the baseline confusion rate. Our claim is that this rate rises with independent membership signals on the source text, and that it does not vanish under multi-step question formats.
Prior work plays three roles here. DyVal supplies the method template: it generates fresh reasoning problems from directed acyclic graphs, varies depth and presentation order, and reports a 13.6% GPT-4 accuracy drop on reachability when the description order is reversed. But DyVal works only in arithmetic, logic and algorithms, and builds new problems from scratch. Because nothing in those problems conflicts with memory, they can measure reasoning but not reliance on memory. CausalProbe-2024 and the arXiv synthesis study Beyond Memorization motivate the work. Both show that freshness matters, but they disagree about why, and neither uses structured event graphs. Representing Implicit And Explicit Time Relations In Narrative supplies the consistency machinery. It represents a narrative as a graph of time points and intervals, then propagates the ordering constraints to expose contradictions; we use this to reject edits that would make the timeline impossible. VALOR-EVAL supplies the evaluation method. It has a language model parse free text into structured tuples and then matches them semantically against ground truth, reaching 82.35% hallucination-detection accuracy against 5.88% for token matching. We apply the same parse-then-match idea to timelines rather than image scene graphs.
Research Program
Source stories and graphs. We start from narratives that come with gold or easily verified event structure and are plausibly memorized. EventStoryLine and Causal-TimeBank news documents give causal links; we add temporal links, with human checks. Choice-75 and proScript-style daily scripts (Choice-75) give branching structure. Story Cloze and e-CARE passages (e-CARE) give short causal chains. CausalProbe-2024 news from early 2024 serves as the low-contamination source pool, and its freshness was already supported by membership-inference testing in the original paper.
Edits. We use four kinds of edit to the graph. The first reverses a single precedence edge. The second deletes a causal edge, keeping in mind that a cause must come before its effect, a constraint ERGO notes it leaves unmodeled. The third inserts a decision branch. The fourth applies a presentation-only change, such as a flashback, a frame story or interleaved threads, which alters the order of telling but not the order of occurrence. After each edit we propagate constraints, and we discard any edit that creates a cycle or an impossible interval.
Rendering and round-trip check. A generator model rewrites the story with the smallest text changes that realize the edited graph. A separate judge model then converts the rendered text back into an event graph. An item is kept only if the recovered graph matches the edited graph on every edge involved in a query. This round-trip check is the main safeguard against the judge's own errors. We audit a stratified sample with human annotators and report agreement between judge and humans before using any scores.
Queries and scoring. Most queries are closed, such as "did X happen before Y?" or "would Y still occur without X?", and their answers are read from the graph by a program, not by the judge. A second set asks the model to reconstruct the timeline in free form. These are scored by parse-then-match against the edited graph, so that a correct paraphrase is not penalized. Each query is written in two forms: a shallow cloze-like form, and a multi-hop form that requires chaining three or more edges.
Main quantity. For a model $M$, let $Q_c$ be the set of queries whose answer differs between the original graph $G$ and the edited graph $G'$. The memory pull is defined as
$$\pi(M) = P_{q \in Q_c}\big(M(q) = a_G(q)\big) - P_{q \in Q_c^{\text{fresh}}}\big(M(q) = a_G(q)\big).$$
Here $a_G(q)$ is the answer under the original graph. $Q_c^{\text{fresh}}$ is the matched set of edited items built from CausalProbe-2024 stories, which have the same edit types and graph sizes but, as far as can be verified, no memorized original. The subtracted term removes the share of "original-answer" errors that come from the edit itself, for example a reversed edge that reads as implausible. Without it, those errors would be counted as memory.
Baselines
The naive reference point is accuracy on the unedited original stories. That is what current benchmarks report, and it cannot separate reading from remembering. We report it for every model so that the drop from original to edited items is visible. The test is whether that drop is concentrated in changed-edge queries and in answers that match the original graph, rather than spread across all queries.
The strongest existing contamination detector is CDD from Generalization or Memorization. It samples many outputs and flags a source as contaminated when those outputs are abnormally similar to one another. Its authors report 21.8% to 30.2% relative improvement in accuracy and AUC over prior detectors on DETCON. We also include Min-K% Prob membership scoring, as used by CausalProbe-2024. Both are run on the same source passages, with the same open-weight models and the same number of samples per item, and both are tuned on the same held-out calibration split. There is no reported result on Lacuna for either detector on narrative event benchmarks, so the comparison is new. To count as beating them, $\pi$ must predict human-verified or influence-verified memorization labels on a held-out subset at higher AUC than either detector. A weaker but still useful outcome is that $\pi$ adds predictive power beyond their scores when combined in a single regression.
On task accuracy, the strongest structure-aware references are ERGO, at 63.9 intra-sentence and 47.1 inter-sentence causality F1 on EventStoryLine (ERGO), and TaIncBC, at 61.9 overall F1 on EventStoryLine and 65.2 on Causal-TimeBank (TaIncBC). We will retrain or run released versions of both on the original and edited versions of the same documents. A fine-tuned classifier that keeps its F1 on edited items would show that relying on the original graph is not inevitable.
For contrast with presentation-order effects, the reference is DyVal's reported 13.6% GPT-4 drop on reachability under reversed order (DyVal). We compare it with our presentation-only edits on narrative text.
Two ablations isolate the key ingredients. The first removes the original story by pairing the same edited graphs with templated DyVal-style text that has no real source. If $\pi$ on real-source items does not exceed the templated rate, the memory interpretation fails. The second removes the round-trip check. This shows how much apparent memory pull is really caused by renderings that did not faithfully express the edited graph.
Why not just…?
Why not just use fresh post-cutoff news, as CausalProbe-2024 does? Fresh data shows that performance falls, with Claude 3 Opus below 70% on CausalProbe-H, but not why, and it stops being fresh after the next pretraining crawl. Rewiring works on text the model has already seen. It identifies which answers came from memory, at the level of single edges, and it is not used up by future crawls, because the conflict between the edited text and the remembered original does not go away. We still use fresh news, as the control term in $\pi$.
Why not just generate synthetic graphs from scratch, as DyVal does, but in natural language? This is a real alternative for measuring reasoning, and we include it as the templated ablation. However, a story nobody has seen cannot trigger the original-answer error, so it cannot measure how much a model leans on memory. If the templated condition gives the same confusion rate as real-source items, this objection wins and the proposal reduces to "DyVal for narratives." The experiment is designed to show plainly whether that happens.
Why not just run CDD or Min-K% on the source texts? These tools say whether a passage looks memorized, not whether memory changes the model's answers to reasoning questions. A passage can be memorized and still be read correctly when edited. We treat them as competitors and as converging evidence. If $\pi$ adds nothing over them in predicting verified memorization, we will say so.
Why not just score answers with exact match or multiple choice and skip the language-model judge? For closed queries we do exactly this, with answers taken from the graph by a program. The judge is limited to two jobs: the round-trip check on renderings, and scoring free-form timeline reconstructions, where exact match fails. MCTACO gives a sense of the scale: BERT reaches 42.7% exact match against 69.9% F1, and humans reach 75.8% exact match. Judge reliability is reported against human audits, not assumed.
Decisive Checks
The idea is supported if three things hold. First, $\pi$ is significantly positive for EventStoryLine, Story Cloze and script sources, and near zero for CausalProbe-2024 sources. Second, $\pi$ rises with Min-K% and CDD scores across passages. Third, real-source items exceed the templated ablation. The disagreement between CausalProbe-2024 and Beyond Memorization is settled by comparing query forms. If multi-hop queries on edited items keep a $\pi$ comparable to cloze queries, synthesis hides memory reliance rather than removing it. If $\pi$ falls to the fresh-source level under multi-hop queries, the synthesis account is right. Presentation-only edits test whether DyVal's order sensitivity carries over to flashbacks and interleaved threads in real stories.
Models will span open weights, including LLaMA 2 for comparability with CausalProbe and so that Min-K% can be computed, and API models evaluated on output behavior only. Each condition will have 500 items per source pool, matching DyVal's per-task scale.
References
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Representing Implicit And Explicit Time Relations In Narrative
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- Choice-75: A Dataset on Decision Branching in Script Learning
- e-CARE: a New Dataset for Exploring Explainable Causal Reasoning
- ERGO: Event Relational Graph Transformer for Document-level Event Causality Identification
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- TaIncBC: Topic-aware In-context Prompt with Bias Calibration for Event Causality Identification
- "Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding