Pausing Where the Story Changes: Latent Compute for State Tracking in Noisy Narratives

Language models read and write stories one token at a time, and every token gets the same amount of computation. Stories are not uniform. Most sentences change nothing important, but a few carry the whole plot: a character moves an object while another is out of the room, or someone learns a secret. Long-form generation fails at exactly these points. Models contradict earlier facts, lose track of who knows what, and get pulled off course by vivid but irrelevant detail in their context. This proposal asks whether a model can be taught to take silent "thinking pauses" where the story's state changes, sized to how much changed, and whether those pauses make it more resistant to distracting context. A pause here is a learnable placeholder token that produces no text but gives the model one more position to compute over.

The two-sentence version is this. We fine-tune a model to insert silent pause tokens right after sentences that change the story's world state, with a small learned head deciding how many pauses each sentence gets. We then test, on program-generated stories whose true state is known exactly, whether this beats the same number of pauses spread evenly, whether it beats writing the state out as text, and whether the benefit grows as irrelevant subplots are mixed into the context.

Core Claim

Our claim is that placement matters more than count. Pause tokens help narrative state tracking under distraction mainly when they sit immediately after state-changing events, and a learned per-sentence budget gets most of that benefit at a fraction of the tokens used by uniform insertion. The measurable prediction is that the accuracy lost as distractor density rises will be smaller with event-anchored pauses than with an equal number of pauses at paragraph boundaries. A finding that placement does not matter would also be informative, and the design is built so that either outcome is clean.

Prior work sets this up without testing it. Learning to Insert [PAUSE] Tokens for Better Reasoning inserts a fixed block of five pauses before tokens the model finds uncertain. It reports gains on GSM8K and MBPP, including +1.82 points on GSM8K for Llama 3 8B, but it works only on short math and code problems, uses a fixed count, and has no notion of an external world state or of distracting context. Pause-Tuning for Long-Context Comprehension inserts pauses after every paragraph and fine-tunes with low-rank adapters (LoRA). It shows the pauses act as attention anchors that flatten the lost-in-the-middle curve, with an average Needle-in-a-Haystack gain of 10.61% for Llama 3.2 3B and 3.57% for Llama 3.1 8B. However, it uses content-blind placement and tests only single-fact retrieval. Both papers motivate the proposal. Neither separates where pauses go from how many there are, and neither tests stories that must stay consistent. DiscoDVT supplies the intuition that long text benefits from latent codes tied to discourse structure, but its codes come at a fixed downsampling rate that cannot scale with scene complexity.

Research Program

The testbed is built on ExploreToM (paper, resource). This program-guided generator writes stories about characters moving objects and observing, or failing to observe, each other's actions. Each story comes with a ground-truth trace of world state and beliefs. We chose it because it removes the evaluation bottleneck that dominates narrative-coherence research: we know exactly which sentences change state and what every character should believe after each one. It is also hard. The ExploreToM authors report 0% for Llama-3.1-70B-Instruct and 9% for GPT-4o on their adversarial action sets.

To study noise sensitivity, we extend the generator with distractor passages at a controlled density, measured as the ratio of distractor sentences to core sentences. There are two kinds of distractor. Neutral distractors are side subplots with other characters and objects. Lures are sentences that mention the tracked objects or characters without changing their state, for example "Anna glanced at the blue box on the shelf." Lures target the failure described in TurtleBench and Less is More: models over-interpret salient but irrelevant narrative detail.

During fine-tuning we insert $k_i$ pause tokens after sentence $i$, where $k_i$ is proportional to the number of state and belief updates that sentence causes according to the generator's trace. As in Think before you speak, the loss is masked on pause positions. At inference no trace is available, so a small classification head reads the hidden state at each sentence-final token and predicts $k_i$. It is trained jointly from the trace labels. This head is the adaptive-depth component. It plays the role of the stopping criterion in Self-Braking Tuning, which relies on natural-language "epiphany" phrases that silent tokens cannot produce.

We train Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct with LoRA. These match the Pause-Tuning setup, and the 8B model matches the ExploreToM fine-tuning setup. Staying in the fine-tuning-only regime directly tests the dispute between Think before you speak, which found fine-tuning-only pauses weak, and the two later papers that found them useful.

We evaluate two tasks. The first is belief and location questions asked after the story. The second is story continuation: the model writes the next paragraph, and a judge model checks it for contradictions against the explicit ground-truth state table. Checking against known facts is a much easier and more reliable job than scoring free-form story quality. We validate the judge against human labels on a few hundred continuations before trusting it. For mechanism, we measure how much attention the answer tokens and the pause tokens place on distractor spans versus the changed-entity span. The goal is to see whether event-anchored pauses concentrate reading on the update rather than spreading it across lures.

Baselines

The naive reference is the same model fine-tuned with LoRA on the same ExploreToM-plus-distractor data with no pause tokens. This matters because ExploreToM fine-tuning alone raised Llama-3.1-8B by 27 points on ToMi and 29 on Hi-ToM (Explore Theory of Mind), so any pause effect must be measured on top of a strong training signal, not on top of a zero-shot model. Beating it means higher accuracy at high distractor density with the gap widening as density grows. A uniform improvement at every density would suggest generic regularization rather than noise resistance.

The strongest existing latent-token method for context noise is Pause-Tuning (paper), with pauses after every paragraph. We run it on identical data and adapters, and its reported numbers are the Needle-in-a-Haystack gains quoted above. For noisy narrative state tracking itself, no reported result was found on Lacuna for any method, so every arm is run by us.

The decisive ablation is a matched-budget variant. Uniform pauses are spread so the total pause count per story equals what our learned head spends. A second arm places pauses at random sentences with the same total. A third places a fixed $k$ at true event sentences, which separates placement from adaptive sizing. The DIT recipe (Learning to Insert [PAUSE] Tokens) is a further baseline. It places pauses by uncertainty rather than by state change, and it tells us whether a generic uncertainty signal already finds the events.

All arms get the same data, the same number of optimizer steps, and the same learning-rate sweep. We report accuracy against added tokens per story, so no arm wins by spending more compute.

Why not just…?

Why not just insert pauses uniformly, as Pause-Tuning does? This is the real threat. Beyond Semantics shows that the content of intermediate tokens can matter surprisingly little, which raises the possibility that only the extra positions matter. The matched-budget uniform and random arms settle it directly. If they match event-anchored pauses at every distractor density, we report that placement is irrelevant for this kind of state tracking. That would be a useful and cheap recipe, and a negative result for the adaptive framing. We will not tune our way around it.

Why not just write the state out as text, as Re3 or LIGHT do? Re3 and the scratchpad in Beyond a Million Tokens both show that distilled explicit state beats raw history for coherence, so an explicit state-summary arm may well win on raw accuracy. Our question is what it costs. The comparison is at matched added tokens: the model writes a one-line state update after each event sentence, trained on the same traces. TurtleBench gives a reason to expect written deliberation to amplify lure sensitivity, since o1-preview's 84.40% trails Claude 3.5 Sonnet's 89.49% on its narrative puzzles. That is only suggestive, and the lure-density curve is where it gets tested. If text summaries dominate at equal budget, the latent approach is not worth pursuing here.

Why not just filter the distractors before the model reads them? Compressors such as CompAct and path pruning such as Superposition Prompting work well for retrieved documents. In a story, however, lures are not separable documents: they share entities with the relevant events, and deciding they are irrelevant is itself the state-tracking problem. We include a sentence-level relevance filter trained on the generator's labels as an upper-reference arm. If a simple filter recovers the clean-context accuracy, the problem is retrieval, not computation, and we would say so.

Proposed Experiments

The primary experiment sweeps distractor density from 0 to 3 distractors per core sentence, separately for neutral distractors and lures. For each arm it reports question accuracy and judged contradiction rate on continuations. The headline quantity is the drop from clean to highest density, plotted against added tokens. Held-out story templates and longer stories than any seen in training test whether the learned budget head generalizes rather than memorizing templates.

Transfer checks run on ToMi and Hi-ToM for belief tracking and on TurtleBench, where the reported baselines range from 73.69% to 89.49%, for narrative lure sensitivity outside the synthetic generator. Needle-in-a-Haystack confirms we have not lost Pause-Tuning's retrieval gains. LongGenBench serves as an exploratory probe of whether event-anchored pauses delay the reported collapse beyond roughly 4,000 generated tokens. We found no numeric baseline for this on Lacuna, so it is reported descriptively.

The results that would change our minds are fixed in advance. If matched uniform pauses equal event-anchored ones, placement does not matter. If the text-summary arm dominates at equal budget, latent computation is not the right tool here. If the budget head's predicted $k$ correlates poorly with true update counts yet still helps, the benefit is not coming from the mechanism we claim, and the attention analysis should show it.