Flagging Story Contradictions by Where Memory Makes the Text Surprising

Long stories written by language models contradict themselves. A character's sister dies in chapter 2 and phones in chapter 9. Current fixes store story facts in a memory, look up the relevant ones before each new chapter, and check the draft against them. That lookup relies on shared words or similar embeddings, and it fails when the earlier chapter said "the conduit fractures were remediated" and the later one says "the broken pipe still leaks." We propose a different check that needs no lookup. Take the model that wrote the story. For the few words around each character or object mention in a new chapter, ask how much more or less likely those words become when the model is also shown the story's memory. If showing the memory makes the actual text much less likely, the passage is probably fighting the story so far. The comparison is done in the model's predictions, not in surface words, so it should survive the vocabulary drift that breaks retrieval.

Language models are also known to react to meaning-preserving rewordings of their prompts. We therefore compute the score under several paraphrases of the memory and flag a passage only when the surprise persists across all of them. A true contradiction should not depend on how the memory happens to be worded; a spurious flag often will. The result is a cheap, localized detector: two forward passes per paraphrase, no generation, and it points at a span of words rather than a whole chapter. It can trigger targeted rewrites inside existing story pipelines.

Core Claim

The claim has two parts. First, a memory-conditioned likelihood contrast, computed only in short windows around entity mentions, detects narrative contradictions when the earlier and later descriptions share few words. Retrieval-based conflict checks and whole-chapter scores miss these cases. Second, requiring the contrast to hold across paraphrases of the memory removes most false alarms. This second part turns prompt sensitivity from a nuisance into a filter.

Let $M$ be a serialized story memory: summaries plus event records of the kind DOME keeps. Let $x_{<t}$ be the current chapter's text before token $t$, and let $w$ be a window of tokens around an entity mention. The local score is

$$s(w, M) = \frac{1}{|w|}\sum_{t \in w}\Big[\log p(x_t \mid M, x_{<t}) - \log p(x_t \mid x_{<t})\Big].$$

A strongly negative value means the memory makes the written words harder to predict. The robust score is $s^{*}(w) = \max_{k} s(w, M_k)$ over $K$ meaning-preserving paraphrases $M_1,\dots,M_K$ of the memory. A window is flagged only if even the most forgiving paraphrase leaves it surprising.

Where This Sits

Three story systems supply the setting and the thing to beat. Re3 plans, drafts from distilled summaries, and edits contradictions through an attribute dictionary; it reports this works for simple character attributes but not for spatial or causal facts. DOC steers each token with a small classifier against detailed outline leaves. DOME stores events as (subject, action, object, chapter) records in a temporal knowledge graph and retrieves them before each chapter, at roughly 200 model calls per story. All three either retrieve memory by surface or embedding match or check only hand-picked attribute types. We check the draft against the whole compact memory through likelihood, so retrieval is not the bottleneck.

The likelihood-contrast idea itself comes from Learning to Reason for Long-Form Story Generation. That paper rewards a chapter plan by how much it lowers the perplexity of the true next chapter. We reuse the contrast with the sign reversed and at the scale of a word window, to detect conflict rather than reward planning.

The windowing choice is constrained by local-context work. Neighboring Distribution Divergence compares masked-model predictions next to an edit instead of comparing embeddings. It reports that cosine similarity is nearly blind to small meaning-changing edits in heavily overlapping text: Pearson 7.8 versus 19.1 on an STS-B synonym/antonym test. Semantic Topic Models found sense evidence peaks in windows of 10–20 words and dilutes beyond that. CDRM re-weights query terms by their immediate neighborhood. These findings motivate scoring windows rather than whole chapters, and they set the prior range for window size.

The lexical-gap motivation comes from retrieval. Yang et al. found many correct answers share no content words with the question. RaFe and InteR close that gap by rewriting or expanding queries. InteR reports drift when expansion is iterated beyond two rounds. We treat query rewriting as a competitor, not a component.

The paraphrase filter is shaped by conflicting evidence on prompt sensitivity. ProSA ties sensitivity to flat, low-confidence output distributions. SENSEL uses prediction flips across prompt rewordings to decide when to abstain, and finds sensitivity negatively correlated with correctness (Pearson $r$ up to −0.40). Flaw or Artifact? argues much measured sensitivity is produced by rigid string-match scoring, and that it largely disappears under semantic judging. Our score is a log-probability rather than a matched string, so it sidesteps that artifact. We use sensitivity in the opposite direction from SENSEL: stability across paraphrases is the condition for raising a flag, not for trusting an output.

Research Program

The first step builds a controlled test bed. We start from chapter sequences in the next-chapter-prediction book corpus of 30 post-2024 books (Learning to Reason for Long-Form Story Generation). Being professionally edited, these books are presumably mostly self-consistent, which gives a realistic false-alarm floor. We inject contradictions into later chapters by editing a fact near an entity mention: a state, possession, location, relation, or a causal outcome. For each injection we produce three versions of the earlier fact in memory, graded by word overlap: verbatim, light paraphrase, and a paraphrase with no shared content words. Paired "consistent paraphrase" edits reword the later mention without changing its meaning. These controls test whether the detector confuses rewording with contradiction.

The second step runs the detector with open-weight models of two sizes, and with the writer model itself where possible. We sweep window size from ±1 to whole chapter and the number of memory paraphrases $K$ from 1 to 8. Memory paraphrases are generated automatically and checked for meaning preservation by an entailment filter.

The third step places the detector inside a story pipeline built on DOME's outline-and-memory loop. Flagged windows trigger a local rewrite of that span only; unflagged chapters pass through. We generate stories from DOME's 20 premises and WritingPrompts premises (Hierarchical Neural Story Generation) at lengths past the roughly 4,000-token point where LongGenBench finds instruction-following collapses.

Baselines

The naive reference is no check at all. We measure how often injected contradictions survive, and the conflict rate of stories written with no revision step.

The strongest existing system is DOME's memory-retrieval conflict check, run with the same backbone, premises, and memory contents as our pipeline. DOME reports a conflict rate of 0.56%, against 1.21% for DOC and 6.78% for plain Llama-3-70B, on 20 premises with Qwen1.5-72B-Chat (DOME). Beating it means lower conflict rate at equal or fewer model calls, and higher detection recall on the zero-overlap slice of the test bed, where retrieval should fail. For contradiction-detection accuracy on injected edits, we found no reported result on Lacuna, so we will establish those numbers ourselves.

The most dangerous competitor is an LLM judge. The same model is shown the full memory and the new chapter and asked to name contradictions. Flaw or Artifact? shows semantic judging is far more stable than heuristic scoring: rank correlation on NarrativeQA rose from 0.40 to 0.87. We give the judge a matched compute budget and tune its prompt over the same number of variants we allow for paraphrasing. Beating it means higher detection AUROC on the zero-overlap slice at matched false-alarm rate, or equal accuracy at substantially lower cost.

We also run a SENSEL-style baseline: the variance of chapter-level outputs across prompt rewordings, used as the flag signal. We add Re3's attribute-dictionary editor and dense-embedding retrieval with RaFe-style query rewriting over DOME's memory.

Three ablations isolate our ingredients: whole-chapter rather than local-window scoring, a single memory wording rather than the paraphrase maximum, and cosine similarity between memory facts and windows in place of the likelihood contrast.

Why not just…?

Why not just ask a strong LLM to check each chapter against the memory? This is the real threat, and it may win on accuracy. Its weaknesses are cost and brittleness. It needs a generation call per chapter, and its verdict can move with prompt wording, which ProSA ties to low-confidence regions. It also returns chapter-level verdicts rather than spans. The test bed settles this directly. If the judge matches our AUROC on the zero-overlap slice at similar compute, the method is not the contribution. We would report the judge as the better tool and keep the lexically graded benchmark as the finding.

Why not just fix retrieval with query rewriting or expansion? RaFe and InteR show rewriting helps single-turn search. But RaFe inherits its reranker's biases, and InteR drifts past two iterations. Neither has been tested on paraphrased story events. Running rewriting over DOME's memory as a baseline shows whether retrieval failure, rather than detection, is what limits the zero-overlap slice.

Why not just put the whole story in a long-context model? LongGenBench finds generation quality collapses past about 4,000 output tokens despite 128K-token context windows. Re3 found raw rolling context gave 14% lower plot coherence than distilled summaries. This alternative helps reading, not writing. Our full-history judge variant tests the reading side directly.

Why not just use output variance across prompts, as SENSEL does? Variance measures uncertainty about the whole output, not conflict with the story. On open-ended generation, it may largely reflect creative freedom. The SENSEL-style baseline on the consistent-paraphrase controls measures exactly this false-alarm rate.

Decisive Checks

The central prediction concerns the zero-overlap slice. There, retrieval-based and cosine baselines should lose most of their recall relative to the verbatim slice, while the local likelihood contrast should lose little. If the contrast degrades just as fast, the core claim is false.

The paraphrase filter should cut false alarms on consistent-rewording controls without lowering recall on true contradictions. If it lowers both equally, it is merely a threshold shift.

Accuracy should peak at window sizes near the 10–20-word range and fall toward whole-chapter scoring. A flat curve would mean locality contributes nothing.

In the full pipeline, the result must show a conflict rate at or below DOME's reported 0.56% with fewer calls per story. Blinded human coherence ratings on stories longer than 5,000 words must not fall, since fewer contradictions bought with blander prose would not count as success.