When a language model writes a long story, it often contradicts itself. A character's eyes are green in chapter one and "the color of a winter sky" in chapter nine. Catching this is hard for two reasons. Standard contradiction checkers (natural language inference, or NLI: models that label a pair of texts as entailing, contradicting, or unrelated) are too expensive to run over every pair of passages. The cheap retrievers used to pick which pairs to check match on words or on topic similarity, and the two contradicting passages frequently share no content words at all. We propose to locate contradictions using a different quantity. We ask how much a given earlier passage makes a new sentence more surprising to a language model, compared with not having seen that passage.
The intuition comes from information-theoretic linguistics. Earlier context normally makes later text more predictable, which is why local word entropy rises through a document as writers compensate (Entropy Rate Constancy in Text). A passage that contradicts a new sentence should reverse this: conditioning on it should raise the sentence's surprisal (its negative log-probability). A language model reads meaning rather than surface words, so this signal should survive vocabulary mismatch. It is also a conditional-probability comparison rather than a cosine similarity, so it does not inherit the transitivity problem that makes embedding retrieval treat "I love dogs" and "I hate dogs" as neighbors. The proposal is a detector built on this signal and a test of whether it holds.
Core Claim
Let $c$ be a short local window preceding sentence $s$, and let $m$ be an earlier memory chunk, such as a paragraph from chapter one. We define the context-induced surprisal gain as
$$g(s, m) = \max_{t \in s} \big[ -\log P(s_t \mid m, c, s_{<t}) + \log P(s_t \mid c, s_{<t}) \big],$$
where $s_t$ is the $t$-th token of $s$ and $P$ is a fixed scorer language model. The maximum over tokens is deliberate: a contradiction usually concentrates on one or two tokens ("blue" versus "green"), and averaging would dilute it. Our hypothesis has three parts. Supporting chunks give $g < 0$, which is the context compensation predicted by entropy-rate constancy. Unrelated chunks give $g \approx 0$. Contradicting chunks give $g > 0$, and this holds whether or not $m$ and $s$ share vocabulary. A single pass that conditions on the whole available history then serves as a cheap gate for whether $s$ conflicts with anything. Per-chunk gains localize the conflicting premise. An NLI model is run only on the few flagged pairs.
The closest existing idea, "use surprisal spikes to trigger NLI," differs in an important way. Raw surprisal under the generator is a poor contradiction signal. The generator sampled the contradicting token, so it assigned that token reasonable probability. Raw surprisal also varies systematically with discourse position: Surprise! Uniform Information Density Isn't the Whole Story shows that surprisal contours follow rhetorical tree structure, with nesting depth the strongest predictor. That paper therefore constrains the design. A raw-spike detector would fire at every rhetorical boundary. Our contrastive difference computes both terms at the same position with the same local context, so these structural effects cancel. Only the effect of the specific earlier passage remains.
Research Program
The work has three stages, each gated by the previous one.
Stage 1: does the signal exist? We measure the sign and size of $g$ on controlled pairs, before building anything. The controlled set has three sources. The first is converted Breaking NLI pairs: the premise becomes $m$ and the hypothesis becomes $s$. These are single-word antonym and co-hyponym substitutions on which classic NLI models collapsed, for example ESIM at 3.3% on the "planets" category. The second is SNLI/MNLI contradiction, entailment, and neutral pairs. The third is a new set of story-embedded pairs. For those, a premise sentence from a WritingPrompts story is rewritten so that the contradicting sentence shares zero content words with it, following the GPT-generated paraphrase and contradiction injection that SparseCL used for MSMARCO and HotpotQA. We vary three factors: scorer model size, distance between $m$ and $s$ (inserting filler paragraphs), and lexical overlap. This separates the "reads meaning" claim from the "shares words" confound.
Stage 2: the detector. Memory chunks are paragraphs. The scorer caches each chunk's key-value states once. Each new sentence is then scored against a chunk with a forward pass over the sentence tokens alone. The whole-history gate runs once per sentence. Per-chunk localization runs only when the gate fires, and NLI verification runs only on the top-ranked chunks. For NLI we use a RoBERTa-large model pre-trained on DocNLI and fine-tuned for three-way labels, since DocNLI itself is binary. The verifier is there to reject false alarms, such as a chunk that makes $s$ surprising because it sets up a genuine plot twist.
Stage 3: closing the loop in generation. We attach the detector to a Re3-style pipeline (Re3) in place of its Edit module. Re3's Edit module compares new passages against a static dictionary of character attributes and misses relational, temporal, and spatial contradictions. When our detector confirms a contradiction, the offending paragraph is regenerated with the retrieved premise placed in the prompt. This tests whether better localization turns into more coherent stories, rather than just a better classifier.
Baselines
The naive reference point has two parts. For detection, it is lexical retrieval (BM25 over memory chunks) followed by the same NLI verifier. For generation, it is Re3 with its original attribute-dictionary Edit module. BM25 is the default practice, and the vocabulary-mismatch literature predicts it fails when answers and queries share no content words (Yang et al.).
The strongest retrieval competitor is SparseCL (Contradiction Retrieval via Contrastive Learning with Sparsity). It scores candidate pairs by combining cosine similarity with the Hoyer sparsity of the difference between their embeddings. It was built specifically for contradiction retrieval under paraphrase and reports an average 11.0% NDCG@10 improvement over standard contrastive baselines on Arguana, MSMARCO, and HotpotQA, while running at least 200 times faster than cross-encoders. We will fine-tune SparseCL on the same training contradictions available to our NLI verifier. It will retrieve the same number of chunks that our localizer passes to NLI, and the same verifier will sit downstream. Beating it means higher recall@k of the true premise on the zero-overlap story set at equal k, and at least parity on the high-overlap set. No reported result was found on Lacuna for SparseCL on long narrative documents.
The strongest brute-force competitor runs the DocNLI-pretrained cross-encoder on every (sentence, chunk) pair. DocNLI reports 64.06 F1 for RoBERTa-large on its test set and 88.84% zero-shot FEVER accuracy. This baseline bounds what the NLI stage alone can reach given unlimited compute. We report its accuracy and its cost in forward-pass FLOPs. Our method wins if it reaches comparable F1 at a small fraction of the compute, or higher F1 because the surprisal ranking resists the lexical failures that Breaking NLI documents.
For natural errors, we compare against the T5 detector from SNaC, fine-tuned on its span-level annotations and evaluated on the InconE category. SNaC reports recall of 0.90 for character errors and 0.82 for scene-transition errors. No reported result was found on Lacuna for its InconE recall, so we will retrain and report it ourselves.
The key ablations isolate the ingredient being claimed. The first replaces $g$ with raw per-token surprisal under the generator, which is the naive "spike" idea. The second replaces $g$ with raw surprisal under the scorer, keeping the model but dropping the with/without contrast. The third replaces the max over tokens with the mean. If raw surprisal matches $g$, the contrastive framing is unnecessary. All methods share the same scorer or encoder budget, chunking, and hyperparameter search grid, tuned on a held-out development split.
Why not just…?
Why not just run a long-context LLM over the whole story and ask it to find contradictions? The file gives direct evidence against this. On BEAM, contradiction resolution was the lowest-performing category for every model evaluated, including when the contradiction lay inside the context window. Structured memory beat vanilla full-history prompting by 3.5% to 12.69% (Beyond a Million Tokens). We still include a prompted long-context model as a baseline, because newer models could close this gap, and we will report it honestly if they do.
Why not just use SparseCL retrieval followed by NLI? This is the real threat. SparseCL already handles paraphrase and non-transitivity. Our bet is narrower: at zero lexical overlap and long distance, a language model's conditional probability carries world knowledge that a trained embedding does not. For example, "winter-sky eyes" conflicts with "green eyes." The distance-by-overlap grid in Stage 1 settles this directly. If SparseCL matches $g$ on the zero-overlap stratum, the contribution shrinks to a cheap gate. We would then reframe the method as SparseCL retrieval gated by whole-history surprisal gain.
Why not just use the generator's own surprisal spikes? A generator rarely finds its own samples surprising. The rhetorical-contour findings of Surprise! also imply many false alarms at discourse boundaries. Locally Typical Sampling keeps token surprisal near the local entropy and reduces repetition, but it has no view of statements hundreds of tokens back. The raw-surprisal ablation tests this explicitly.
Why not just run NLI on all pairs? Cost grows with the number of sentences times the number of chunks, with full pair length each time. Sentence-level NLI is also brittle under exactly the lexical substitutions that appear across chapters (Breaking NLI). The all-pairs baseline measures both the accuracy ceiling and the compute gap.
Proposed Experiments
Decisive check one: the sign test. On converted Breaking NLI pairs and zero-overlap story pairs, we report the distribution of $g$ for contradiction, entailment, and neutral pairs, and the area under the ROC curve for separating contradictions from the rest. If contradictions do not produce reliably positive $g$ at zero overlap, the project stops or pivots. The ROC curve should reach well above chance on the antonym and co-hyponym categories, where classic NLI failed.
Decisive check two: long-range localization. On injected-contradiction stories spanning 2,500 to 7,500 words, the length range Re3 produces, we report recall@1 and recall@5 of the true premise chunk. These are broken down by distance and lexical overlap and compared against BM25, dense retrieval, and SparseCL. The same analysis runs on BEAM's contradiction-resolution questions. Natural errors are evaluated on SNaC InconE spans with span-level F1 against the SNaC T5 detector.
Decisive check three: downstream coherence. Re3 with our detector-driven Edit module is compared against original Re3 on WritingPrompts premises. Blind human judges rate overarching plot coherence and count contradictions per story. Re3 reports a 14% absolute gain in coherent overarching plot over its baselines, which sets the scale of improvement worth claiming. We also report detector compute per 1,000 words, so the efficiency argument rests on measurement.
A negative result is still informative. If context-induced surprise fails to track contradiction, that is evidence that language models encode story state in a way their next-token probabilities do not express. That finding bears on every proposal that treats model likelihood as a proxy for consistency.
References
- Entropy Rate Constancy in Text
- Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse
- Breaking NLI Systems with Sentences that Require Simple Lexical Inferences
- Contradiction Retrieval via Contrastive Learning with Sparsity
- DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
- Re3: Generating Longer Stories With Recursive Reprompting and Revision
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- SNaC: Coherence Error Detection for Narrative Summarization
- Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
- Locally Typical Sampling