Executing the Story: Interpreter-Checked State Memory for Long-Form Generation

Language models writing or reading long texts lose track of what is true in the world they are describing. A character who left the room is back without re-entering. A user's stated budget changes silently. A report contradicts its own earlier section. Two reasons stand out. The model has no explicit record of the current state of the world. And the memory systems added to fix this store text or text-like notes, then hope that similarity search will bring back the passage that conflicts. This proposal takes an old idea from linguistics called procedural semantics: the meaning of a sentence is a small program that changes the listener's mental state. We apply it as a working memory for large language models. Each passage, whether read or generated, is compiled by the model into a few typed state-update operations, such as introducing an entity, setting a single-valued attribute, moving an object, or recording that an agent learned something. A deterministic interpreter then executes them. Because the interpreter keeps the current value of every variable and a pointer to the text span that set it, a contradiction is detected at the moment it is written, as a failed precondition or an overwrite without a change event. The system does not have to find it later through retrieval.

In two sentences: store long text as the execution trace of a state-update program rather than as chunks or summaries. Then contradictions become interpreter events caught at write time, and retrieval fetches the source span behind each live variable the next sentence touches, instead of fetching passages that merely look similar. This targets the capability that current long-memory systems handle worst. It also tests a claim the literature has never isolated: that executability, not just structure, is what closes the gap.

Core Claim

Retrieval-augmented generation (RAG) conditions a model on passages fetched from an external store. Its main failure in long-form settings is relational, not factual. Models get facts right and invent the connections between them. Groundedness in Retrieval-augmented Long-form Generation audits long-form answers on ASQA and finds that 22.7% of sentences in correct Pythia-12B answers are grounded in neither retrieved documents nor pretraining data, and that Falcon-180B still fails to ground roughly 25% of its correct output. Beyond a Million Tokens shows on its BEAM benchmark that Contradiction Resolution is the lowest-performing memory category across every model and method, including their own three-tier memory, LIGHT. Our claim is that this category stays low for a structural reason. When contradiction checking is deferred to question time, the conflicting earlier statement has to be retrieved by surface similarity, and the two sides of a contradiction are often not similar ("she sold the car" versus "she drove to work"). An executable state removes the search problem. The earlier statement has already written a value to a variable, and the later one either overwrites it through a legitimate change operation or triggers a conflict.

The claim is falsifiable and narrow. It predicts gains concentrated on contradiction, state-update and event-ordering probes. It predicts little or no gain on single-hop fact recall. And it predicts that these gains disappear when the same state is kept as free-text notes instead of executed.

What Prior Work Does and Where This Differs

Classical procedural semantics supplies the conceptual method. Early work treated messages as programs over state machines (COLING 1969 preprint), and Bien formalised utterances as instructions that modify a listener's "multiple environments" (Toward A Multiple Environments Model Of Natural Language). Translation into dynamic predicate logic, a logic in which sentences pass variable bindings forward the way a program passes variables, recasts anaphora and presupposition as variable binding (Bouche et al.). The spatial-language work built on Incremental Recruitment Language executes meaning as networks of cognitive operations against scenes (Robust Natural Language Processing). These systems also constrain the design. They worked only in micro-worlds with hand-built grammars, and scaling to unconstrained multi-clause text is their stated open problem. We therefore do not attempt full compositional semantics. We use a small generic operation vocabulary, let the language model do the parsing, and leave anything that does not compile as ordinary text, reporting how much that is.

Three recent systems add structure to long-context memory and are the closest competitors. SEEM converts interaction streams into episodic event frames (participants, action, time, location, reason), merges redundant frames, and uses provenance pointers to recover full context. It reports 61.1 F1 on LoCoMo and 65.0% on LongMemEval. Re3 keeps an attribute dictionary of character traits during plan–draft–rewrite–edit story generation, and its own human study found the edit stage mattered less than the rewrite stage. LIGHT in Beyond a Million Tokens combines episodic vector retrieval, a working buffer and a compressed natural-language scratchpad. All three store declarative records populated by prompting, and none executes them. A SEEM frame saying that Alice sold the car sits next to a later frame in which Alice drives the car, and nothing in the system notices. Our proposal differs in one respect that we can isolate experimentally: records are operations with preconditions and effects, run by an interpreter that keeps current values and raises conflicts.

Two further papers supply method. Explore Theory of Mind writes stories in a hand-coded domain language with programmed transitions for locations and asymmetric beliefs. It shows that frontier models collapse on the resulting tracking questions (Llama-3.1-70B at 0% and GPT-4o at 9% on complex action sets). It uses the language only offline to generate adversarial data. We reuse its idea of belief-scoped state as an inference-time memory, and we use its gold programs as supervision for measuring our compiler. SEM-CTRL enforces answer-set-programming constraints during decoding and reaches 100% validity on puzzles and planning, but it notes its binary-validity approach does not fit open text. We keep the symbolic check but move it from the token level to the passage level, where the output of the check is a signal to revise rather than a hard mask. R-WoM grounds an agent's predicted world transitions in retrieved tutorials for GUI tasks. We make the analogous move for narrative state, where the "world model" is the interpreter and retrieval serves its variables.

Finally, When Splitting Makes Stronger proves that divide-and-conquer prompting helps only when subtasks are nearly independent. That is a negative constraint for narratives, where each section depends on state changed by earlier ones. It motivates carrying an explicit state between sections rather than generating sections in isolated contexts.

Research Program

The system has three parts.

The first part is a compiler. A language model reads a passage (a dialogue turn, a retrieved document chunk, or a freshly drafted paragraph) and emits operations from a fixed vocabulary of roughly a dozen types. These include introducing and aliasing entities, setting and changing single-valued attributes, adding members to multi-valued attributes, moving an entity between locations, starting and ending events with coarse timestamps, and recording that an agent observed or was told a fact, which scopes the fact to that agent's belief state as in ExploreToM. Attribute keys are free-text but canonicalised by embedding against keys already in use. The model also declares whether each attribute is single-valued, and it can revise that declaration.

The second part is the interpreter, which is deterministic code. It maintains the current world state and one belief state per agent. It checks each operation's preconditions: an entity must exist before it is moved, and a single-valued attribute may change only through an explicit change operation or a time step that permits change. On every write it stores a pointer back to the source span. When a precondition fails, the result is a conflict record containing both source spans.

The third part is state-keyed retrieval. Before drafting the next section, or when answering a question, the model states which entities and attributes it will touch, and the retriever returns the provenance spans of their current and previous values. Ordinary similarity retrieval fills the rest of a fixed context budget.

For reading tasks, conflicts are stored and surfaced when a question touches the conflicting variables. For writing tasks, a drafted section is compiled and run before it is committed. If it raises a conflict, the model receives both spans and either revises the draft or adds an explicit change event ("she bought the car back"). The interpreter never rewrites text itself. It only decides what the model must look at.

The analysis half of the program asks where the approach breaks. We measure compiler accuracy against gold programs on ExploreToM. We measure coverage, meaning the fraction of sentences that yield at least one operation, on BEAM dialogues and WritingPrompts stories. And we measure the false-conflict rate, meaning flagged conflicts that humans judge to be legitimate change, figurative language or perspective shifts. The construal literature ((Re)construing Meaning in NLP) predicts that perspective and metaphor will cause false conflicts, so we annotate them as a separate error class rather than hide them in an aggregate.

Baselines

The naive reference point is vanilla long context: the full history or document set placed in the prompt of the same model, with no memory structure. Beyond a Million Tokens shows this degrades sharply with length, and that LIGHT gives up to +155.7% over it at 10M tokens for some open-source models. We therefore expect it to lose, but it anchors the scale. Standard chunked similarity RAG with a cross-encoder reranker is the second reference point.

On memory QA, the strongest existing methods are LIGHT on BEAM and SEEM on LoCoMo and LongMemEval. For LIGHT, the file reports only relative gains of 3.5% to 12.69% over the strongest baselines and no absolute Contradiction Resolution score, so we will rerun it. For SEEM the reported figures are 61.1 F1 on LoCoMo overall, 54.6 F1 on LoCoMo temporal reasoning, and 65.0% on LongMemEval (SEEM). These are the most informative competitors because both already add structure and provenance. If executability matters, it has to beat structure without execution.

On generation, the strongest competitor is Re3 (Re3). It reported a 14% absolute increase in stories judged to have a coherent overarching plot versus rolling-window GPT-3 on WritingPrompts. On long-form attributed answering we include intent-aware generation (Improving Attributed Long-form Question Answering with Intent Awareness), which reports +2.9 points for frontier models on SQA-CS-V2.

All baselines use the same backbone model, the same retrieval corpus and embedding model, the same total context budget per call, and a matched budget of model tokens per input token processed. We give baselines the extra calls our compiler consumes as additional retrieval or summarisation rounds, and each method gets the same small development set for prompt tuning.

Two ablations isolate the key ingredient. The notes ablation keeps our compiler and state-keyed retrieval but stores the state as a natural-language scratchpad that the model reads and updates, with no interpreter. This is the LIGHT/Re3 design with our extraction prompts. The similarity ablation keeps the interpreter but retrieves by embedding similarity only. Beating the published systems counts only if we beat them on BEAM Contradiction Resolution and on LoCoMo temporal reasoning with overlapping bootstrap confidence intervals excluded, while staying within noise on single-hop recall. Beating the notes ablation by a clear margin on the same categories is the result that would support the core claim.

Why not just…?

Why not just use a million-token context window? The evidence says long inputs do not produce working memory. Vanilla models degrade as BEAM conversations lengthen (Beyond a Million Tokens). On the output side, LongGenBench finds that instruction adherence falls sharply after about 4,000 generated tokens, even for models with 128K-plus windows. Our vanilla long-context baseline reproduces this at each length bucket, so the comparison is made directly rather than assumed.

Why not just keep a natural-language state summary, like LIGHT's scratchpad or Re3's dictionary? This is the most serious threat. A strong model maintaining careful notes might catch most conflicts, and the interpreter might add only brittleness. Re3's weak edit stage and LIGHT's low contradiction scores suggest passive notes fall short, but neither isolates the question. The notes ablation does: same extraction, same retrieval, same budget, with execution removed. If that ablation matches the full system, we will report that executability is unnecessary and that the gain came from state-keyed retrieval, which is itself a useful and simpler finding.

Why not just run a contradiction checker (an entailment model or a model-as-judge) over retrieved prior passages after each draft? This fails exactly when retrieval misses the conflicting passage. Our hypothesis is that similarity retrieval usually does miss it, because the two sides of a contradiction share few words. We test this by adding a post-hoc verifier baseline with a matched call budget and by logging, for each missed contradiction, whether the conflicting source span was ever retrieved. If the verifier's misses are not explained by retrieval misses, our hypothesis about why contradictions are hard is wrong.

Why not just fine-tune the model on state-tracking data? Explore Theory of Mind shows fine-tuning Llama-3.1-8B on its data yields +27 points on ToMi and +29 on Hi-ToM, so this is a real alternative for short stories. It does not address stories whose state far exceeds what a single context holds. We include the fine-tuned model as a backbone in both the plain and interpreter conditions, to test whether the two gains add up or overlap.

Proposed Experiments and Decisive Checks

The first experiment validates the compiler on ExploreToM (dataset), where gold state programs exist. We report operation-level precision and recall against gold, and question accuracy on the adversarial action sets where GPT-4o scored 9%. Answers come from the interpreter's state rather than from the model reading the story. If compiled-state accuracy is not far above direct answering, the downstream program is not worth running.

The second experiment is memory QA on BEAM, LoCoMo and LongMemEval. We report per-category scores and focus on Contradiction Resolution and Event Ordering (BEAM) and temporal reasoning (LoCoMo), against SEEM, LIGHT and both ablations.

The third experiment is long-form generation. We generate stories of at least 2,500 words from WritingPrompts, following Re3's protocol, and long reports on SQA-CS-V2. We measure the contradiction rate by having annotators, blind to condition, audit pairs of sentences that mention the same entity. We also measure groundedness of connective claims, using the sentence-level attribution protocol of Groundedness in Retrieval-augmented Long-form Generation, and pairwise human judgments of overall plot coherence against Re3. As a secondary stress test we run LongGenBench's temporal scenarios, where periodic constraints can be encoded as interpreter state. The file gives no reported absolute STIC numbers, so baselines there will be our own reruns.

The decisive check is a controlled injection test. Into BEAM-length dialogues and generated stories we plant contradiction pairs at varied distances and with varied lexical overlap between the two statements, together with matched legitimate state changes. The prediction specific to this proposal is that detection by similarity-based systems, including the post-hoc verifier, falls as lexical overlap falls, while interpreter-based detection stays flat across overlap and fails only when the compiler misparses. We measure the false-conflict rate on the legitimate changes. If our system's detection also falls with overlap, or its false-conflict rate on legitimate changes exceeds a level at which revision harms fluency in human judgments, the core claim is refuted.