Is Forgetting Inherited? Prerequisite-Linked Evaluation of Knowledge Updates

When a language model is fine-tuned on news about events it has never seen, it often seems to learn the gist and still fail at multi-step questions about those events. We do not know why. One possibility is that the model never stored the new atomic facts correctly: an exact number, a date, or the fact that an old value was replaced rather than erased. In that case the multi-step failure is inherited from a low-level error. The other possibility is that the facts are stored and the model cannot combine them. The two explanations call for different fixes. The first is a data-fidelity problem. The second is a training-objective problem. Current benchmarks report one accuracy per question type, so they cannot tell the two apart.

We propose to separate them. We will build an evolving synthetic world in which every hard question is linked, instance by instance, to the easier questions it depends on. The training text that teaches the model about this world is written by a language model under human-authored templates. A program then checks every document against the world's ground truth before the model trains on it. With this setup we can measure how many post-update reasoning failures are inherited from atomic failures. We can also test whether faithful, verified update data removes them. The human contribution is concentrated where it is cheap and durable: writing templates and validity rules once, rather than reviewing every generated document.

Core Claim

The claim has two parts. First, after parametric knowledge updating (fine-tuning the model's weights rather than giving it retrieved text), most failures on compositional and ripple questions are inherited. They occur on instances where the model already fails a prerequisite atomic or temporal-relation probe, especially probes about exact numbers and about what was true before the update. Second, update data whose facts are fixed by a versioned world graph and checked by round-trip extraction will raise atomic accuracy and, through it, compositional accuracy. Free-form LLM-written news at the same token budget will not.

If the second part fails, so that atomic accuracy rises but compositional accuracy does not, that is an equally informative result. It would place the bottleneck in how the model combines stored facts rather than in the data.

Where This Sits

The closest work is Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs. It builds 41 fictional 2030–2035 event graphs (PARALLELEVENTS), has GPT-4 expand them into roughly 1,000-word news articles, and updates models with continued pre-training plus preference optimization (SYNAPSE). It reports over 80% causal-reasoning accuracy. It also reports a "numerical gap": exact values such as a wind speed of 132 knots are not retained. That paper motivates ours and supplies the world graphs and the update method we reuse. We differ in three ways. Our world is versioned, so values change over time and old values become obsolete rather than absent. Our generation is bounded by templates and checked by a program. Our evaluation links each multi-hop question to its prerequisite probes, whereas that paper reports flat task and causal accuracy.

Three other lines of work constrain the design. History Matters: Temporal Knowledge Editing in Large Language Model shows that standard weight-editing methods drive historical-fact accuracy to near zero. Its METO method raises HES* on AToKe-ME only from below 0.3 to above 15. Any update method therefore has to be scored on old states as well as new ones, and our hierarchy includes historical probes at every level.

EvolveBench organizes temporal competence into five dimensions on real Wikidata-derived facts and documents a roughly 48% drop under timestamp-misaligned context. Its dimensions are parallel categories, not per-instance prerequisites, and it does not update models. We use it only as a check that our updates do not damage general temporal awareness.

MQuAKE-Remastered found that up to 76% of MQuAKE questions or labels were corrupted. That is a direct warning that unchecked generated benchmarks mislead, and it is why our gold labels come from program execution over the graph rather than from model output.

The human-in-the-loop design follows ToolQA and EHRSQL. ToolQA had humans write 117 templates and computed answers with code. EHRSQL turned clinician questions into 230 templates with explicit time expressions. Both used this approach to build evaluation sets. We apply it to building training data for knowledge updates, which neither did. The idea of compressing a source into structure before generating text echoes CLIPPER, where generating from outlines cut claim errors from 73.1% to 16.7%. CLIPPER worked on static books and had no verification step.

Research Program

Versioned world. We extend PARALLELEVENTS so that a subset of entity attributes changes at a second and third time step. Examples include a stadium's capacity after repairs, a federation's president after an election, and a revised casualty count. Each fact therefore carries an explicit validity window. Old values become superseded facts that the model must still answer correctly when the question asks about an earlier time.

Templates and validity rules. Annotators with no special domain expertise write about 100 templates in three tiers.

  • Level 1, atomic: the value of an attribute at time t, including exact numbers and dates.
  • Level 2, temporal relations: which value held at a given time, which came first, and how long it lasted.
  • Level 3, composition and ripple: multi-hop consequences that require Level 1 and Level 2 facts.

Annotators also write the rules that determine which consequences follow from which changes. Every Level 3 instance is generated together with pointers to the specific Level 1 and Level 2 instances it depends on. Answers at all levels are computed by executing the template against the graph.

Training text. A language model turns slices of the graph into news-style articles and timelines. Each document is constrained by a template naming which facts, with which values and timestamps, must appear. A separate extraction pass reads the values back out of the document and compares them with the graph. Documents that fail this round-trip check are regenerated. Humans audit a random 5% sample. The audit serves two purposes: measuring the residual error rate, and measuring human minutes per verified document, which we report as a cost.

Diagnostic quantity. Let $E_3$ be the set of Level 3 errors on the evaluation set. The inherited fraction is $|E_3 \cap F|/|E_3|$, where $F$ is the set of Level 3 instances with at least one failed prerequisite probe. We also report Level 3 accuracy conditioned on all prerequisites passing, which measures pure composition ability. These two numbers, broken down separately for current-state and historical-state questions, are the main outputs of the study.

Baselines

The naive reference point is the unupdated base model, an open 7–8B model, evaluated on the same hierarchy. It should answer near chance on post-event facts. It tells us how much of any measured score comes from priors rather than from the update.

The strongest existing parametric method is SYNAPSE from Synthetic Worlds. It is the most direct competitor because it already updates models from LLM-written news about the same synthetic events and reports over 80% causal accuracy and a 14.23% multi-hop gain over MeLLo on PARALLELEVENTS. We will re-run its pipeline on our versioned world with the same base model, the same number of training tokens and epochs, and the same hyperparameter search budget as our method. Its documents will be generated free-form from the same graph slices. Beating it means higher Level 1 numeric and historical accuracy and higher Level 3 accuracy at matched tokens. Matching its causal accuracy on the original PARALLELEVENTS questions while losing on our historical probes would not count as a win.

For historical retention we include METO from History Matters, run on the triples of our versioned graph. Its reported HES on AToKe-ME is "over 15". We will also report all of our update methods on AToKe-ME so the comparison sits on an established benchmark. Beating it there means a higher HES at comparable current-edit success.

As a non-parametric reference we include TEMPLE-MQA from Multi-hop Question Answering under Temporal Knowledge Editing, a time-aware retrieval graph that reports HES 97.25 on AToKe-SE, and GWalk, which reaches 57.37% on cleaned MQuAKE-CF-3K. We do not expect parametric updating to beat them on raw accuracy. They show how far weight updates remain from structured memory, and our hierarchy can show whether their remaining errors are also inherited.

Three ablations isolate our ingredients, all at matched tokens:

  • our pipeline with the round-trip verification step removed;
  • our pipeline with LLM-written templates in place of human-written ones;
  • SYNAPSE-style free-form data rewritten so that each fact is mentioned exactly as often as in our data.

Why not just…?

Why not just use retrieval or a graph memory instead of updating weights? For accuracy on the current state, retrieval may well win. TEMPLE-MQA's 97.25 HES on AToKe-SE is far above any parametric number in the file. Retrieval still has its own failures: SituatedQA found retrieval systems 15 points worse on updated facts than on stable ones, and EvolveBench found roughly 48% drops under misaligned timestamps. More to the point, our question is about why parametric updates fail, and a retrieval layer does not answer it. We report retrieval as a reference rather than a competitor, and we apply the same inherited-error analysis to it.

Why not just have a stronger model generate the news and a model judge check it, with no humans? This is a real threat, because the humans in our design do not review documents. If LLM-written templates combined with round-trip checking match human templates, the human role reduces to the 5% audit. The template-source ablation settles this directly. We expect human templates to matter mainly at Level 2, where the rules about which value holds over which window are easy for a model to specify loosely. If they do not matter, we will report that and recommend the fully automated variant.

Why not just repeat the exact numbers more often? The numerical gap reported in Synthetic Worlds may simply reflect how rarely each number appears in the training text. The frequency-matched ablation tests this. If frequency-matched free-form text closes the Level 1 gap as well as verified templated text does, then verification is unnecessary for atomic facts. The remaining question would be whether it still helps historical-state probes, where repeating the new value can actively overwrite the old one.

Decisive Checks

The study rests on two measurements.

The first is the inherited fraction of Level 3 errors under SYNAPSE-style updating. If it is high, say above two-thirds, the low-level hypothesis is supported. If Level 3 accuracy conditioned on all prerequisites passing is itself low, composition is the bottleneck and better data is the wrong fix.

The second is whether the Level 1 gains from our pipeline carry through to Level 3, for current-state and historical-state questions separately, at matched tokens. We will also check for side effects. General temporal awareness, measured by EvolveBench against GPT-4o's reported 79.36 exact match as a reference ceiling, should not degrade. Held-out math and coding performance should not degrade either; Synthetic Worlds reported a drop there, and no specific benchmark numbers for it were found on Lacuna. Human minutes per verified document will be reported alongside every accuracy number, so the cost of the human contribution is visible.