Checking Player Actions Against Story History: Multi-Hop Retrieval for LLM Game Masters

When a language model runs an interactive story as its narrator or "game master," every player action forces a quiet decision. The model can go along with the action, refuse it, or invent a reason it makes sense. Current models make this decision badly in both directions. Models tuned to be helpful tend to accept player claims that contradict the established story. In Can LLM Agents Stick to the Script? Modeling Commitment in Interactive Narratives, even GPT-5.2 keeps its story intact in only 40% of sessions after 20 adversarial turns. Systems built to prevent this, such as the scene-by-scene planner SNAP, hold the story together but feel restrictive to players. Our proposal treats this decision as a retrieval problem. Before responding, the narrator searches the story's own recorded history, hop by hop, for a chain of events that either licenses the player's action or rules it out. It then accepts, refuses, or accommodates the action according to what that chain shows.

The reason this matters is that the two failures share a cause. Whether an unexpected action is legitimate often depends on facts several steps back. For example, the player says they unlock the vault. That depends on whether they took the key from the guard. That depends on whether the guard survived the ambush three scenes earlier. Answering such a question is exactly the multi-hop dependency chain that iterative retrieval methods were built for in question answering. Yet those methods have never been pointed at the interactive narrator's problem: deciding when to say yes without breaking the story.

Core Claim

Interactive-narrative work in the planning tradition already names the right test. Problem Formulation for Accommodation Support in Plan-Based Interactive Narratives argues that an exceptional player action should be accommodated when the new state can be causally explained from what came before, which it calls "backward inferences." That work implements the test with hand-authored planning domains. LLM narrators skip the test entirely.

Our claim is that an LLM narrator can run this backward-inference test by interleaving reasoning with retrieval over an append-only log of what has actually happened in the story. Doing so should lower two error rates at once. The first is false acceptance: yielding to actions that contradict commitments. The second is false refusal: blocking actions that history actually supports, which is the railroading complaint against SNAP. We further predict that the advantage over simply putting the whole history in context grows with the number of hops the justification requires. That prediction follows from NovelHopQA, which found that hop depth, more than context length, drives multi-hop failure in long narratives.

What is new is the target of retrieval and the way success is scored. In IRCoT, ITER-RETGEN, and DRAGIN, retrieval answers a fixed factoid question over Wikipedia. Here retrieval adjudicates a player's intervention against a world that keeps changing. We also score the narrator on both sides of the tradeoff between player freedom and story coherence, not on one accuracy number.

Research Program

The system has three parts.

The first is the story log. Each realized scene is stored as a set of short, entity-normalized event records: who did what, where, and when, plus any state changes. These records sit alongside the environment's fact ledger and commitments, using the NCP-Bench format. We resolve entity aliases when records are written, because coreference confusion across chapters is one of the failure modes NovelHopQA identifies. The log contains only events that have already happened. Future plot milestones are visible to a separate commitment checker, never to the text generator. This addresses the "script leakage" seen in Towards Enhanced Immersion and Agency for LLM-based Interactive Drama, while avoiding the scene-isolation amnesia that SNAP's cells impose.

The second part is the justification search. Given a player action, the narrator first writes out what must be true for the action to be possible. It then runs an IRCoT-style loop for each precondition. The model writes one reasoning step, uses that step as the next search query, and continues until it reaches one of three outcomes: a supporting chain, a contradicting record, or nothing relevant. We cap the search at a fixed number of hops so that compute is matched across methods.

The third part is the verdict policy. If the history supports the action, the narrator accommodates it. If the history contradicts it, the narrator refuses within the story world and grounds the refusal in the cited record, rather than giving an out-of-character "you can't do that." If the history is silent, the action introduces a genuinely new fact. In that case the narrator checks the new fact against the retrieved commitments, and if nothing breaks, it accepts the fact and writes it into the log as a new commitment. This silent case is the agency lever. It lets players author unscripted facts while ensuring those facts bind the rest of the session.

Baselines

The naive reference point is a vanilla narrator with the NCP-Bench system prompt, the fact ledger, and a rolling transcript. On NCP-Bench, the cited paper reports a 40% survival rate after 20 turns for GPT-5.2, survival "near zero" for DeepSeek-V3.2, Qwen3, and Grok-4.1, and fact conflicts in "40% to 68%" of interactions. A human narrator reaches 100%. These numbers are the floor and ceiling we must place ourselves between.

The strongest competitor is a long-context self-check narrator. It receives the full ledger and full uncompressed history, and it is prompted to perform the same three-way verdict before responding, but it does no retrieval. This is the baseline most likely to match us, because it isolates whether retrieval adds anything beyond asking the model to check its own work. NovelHopQA bounds what such a narrator can do: even with golden context, accuracy falls "about 12 points" from 1 to 4 hops at 64k tokens, and Gemini 2.5 Pro and o1 "dipped below 80%" on 4-hop questions at 128k. We give this baseline the same model, the same verdict instructions, and a matched token budget per turn. Beating it means lower false acceptance at equal or lower false refusal, at hop depths of 3 or more, with a confidence interval that excludes zero.

We also run three other external baselines. The first is HiAgent-style compressed memory, which the NCP-Bench authors found increases local errors. The second is one-shot retrieval of the top-scoring chunks from the log; NovelHopQA reports that a similar setup with roughly 2.5k tokens of retrieved context lands "around 50%", a 25–35 point drop against golden context. The third is DRAGIN, which decides when and what to retrieve from internal model signals and reports 0.3931 F1 on 2WikiMultihopQA with LLaMA2-13B. Because DRAGIN needs access to model internals, we run it only on open-weight models.

As the coherence-maximizing anchor, we include a SNAP-style planner that refuses anything off the current scene plan. SNAP reports 4.510 automatic continuity on Wikiplot, against 3.819 for vanilla GPT-4o. We expect this planner to score well on coherence and poorly on false refusal.

The key ablation keeps our full pipeline and changes one thing: the justification search is limited to a single retrieval round. Any gain over this ablation is attributable to multi-hop retrieval specifically, rather than to the verdict policy or the event log. A second ablation provides the gold evidence chain as an oracle, which bounds the gain available from retrieval alone. All methods share backbone models (Qwen3 open-weight and GPT-5.2), adversarial player scripts, per-turn token budgets, and an equal prompt-tuning budget on a held-out development split.

Why not just…?

Why not just put the whole history and ledger in a long context window? This is the real threat. Over a 20-turn NCP-Bench session, the history may fit comfortably in context, and a careful self-check prompt may match retrieval. We do not argue this away. The hop-stratified experiment below settles it. Our prediction, drawn from NovelHopQA's finding that hop depth hurts even with golden context, is parity at 1–2 hops and a growing gap beyond that and over longer sessions. If parity holds at every depth, then the bottleneck is the decision policy, not access to evidence, and we will report that the retrieval component is unnecessary.

Why not just compress memory with hierarchical summaries? The NCP-Bench authors tested exactly this with HiAgent. They found that compression "strips away the nuance of local player intent" and raises local errors. Justification chains also depend on specific small facts, such as who holds the key, which are the details summaries discard. We include HiAgent mainly to confirm this result on our probe set.

Why not just add a critic that checks each draft response against the ledger? A one-pass critic catches direct contradictions of atomic ledger facts. It cannot catch an action whose illegitimacy only appears after chaining several events, and it does nothing for the false-refusal side of the tradeoff. The single-round ablation is effectively this critic with retrieval. If it matches the full system, multi-hop search is not earning its cost.

Why not just tighten the scene plan, as SNAP does? Tightening raises coherence by refusing more. SNAP's own annotators said it felt restrictive. We expect a SNAP-style planner to sit at low false acceptance and high false refusal, and the two-sided metric makes that cost visible instead of hiding it.

Proposed Experiments

The main experiment is a hop-stratified probe set built on top of NCP-Bench environments. For each of the 100 environments, we play out a reference session. Then, using NovelHopQA's anchor-chaining construction, we generate player actions whose legitimacy depends on chains of 1 to 4 logged events. By construction, half of these actions are supported by history and half are contradicted. We add a third class of "silent" actions whose new fact either breaks no commitment or breaks one. Every probe therefore has a known correct verdict.

On this probe set we report false-acceptance and false-refusal rates as functions of hop depth and of session length, at 20, 60, and 120 turns. The central figure plots each method's two error rates. A method is better if it moves toward lower error on both axes, not just along the tradeoff between them.

We also report the standard NCP-Bench metrics under its adversarial player: survival rate, fact conflicts, and commitment violations. As a component sanity check, we run the justification search as a plain question-answering retriever on NovelHopQA. This tests whether it narrows the reported 25–35 point gap between retrieval and golden context.

Finally, we run a human study. Participants play sessions with our narrator, the long-context self-check narrator, and the SNAP-style planner. Each session contains choices designed, following Foreseeing Meaningful Choices, to produce real shifts in time, place, cause, or character goals. Participants rate perceived agency and coherence. Following Towards a theory of choice poetics, we separately log actual influence on story state, because players can feel in control without changing anything.

Three outcomes would be decisive. First, the full system should beat the long-context self-check narrator on false acceptance at 3–4 hops without raising false refusal. Second, it should beat the single-round ablation. Together these show that multi-hop retrieval, and not the verdict prompt, is doing the work. Third, human agency ratings should be above the SNAP-style planner while coherence ratings are not below it. If the first check fails at every depth, the honest conclusion is that LLM narrators fail at commitment because of how they decide, not because they cannot find the evidence, and the research should move to training the verdict policy.