Web-Weighted Character Signatures for Detecting Identity Drift in Long Stories

Long stories written by language models tend to lose track of their characters. A cautious archivist in chapter one starts talking like a swaggering smuggler by chapter six, or quietly takes over another character's habits and relationships. Current fixes either write down a few explicit facts per character, such as age and profession, and check them, or ask a model to reread the whole story. The first misses most of what makes a character a character. The second is exactly the long-range contradiction task that current long-context models do worst.

We propose a cheap, inspectable alternative. For each character, collect the short phrases that surround their mentions, with the name itself removed ("masked"). Then weight each phrase by how rare it is in a web-scale n-gram corpus, meaning a table of counts for every short word sequence seen across billions of web words. Phrases that are common on the web ("walked into the room") say nothing about who a character is. Phrases that are rare on the web but recur around one character ("adjusted her stethoscope", "muttered a sailor's curse") make up that character's signature. Comparing signatures across chapters, and across characters, gives a document-level test for identity drift that is built entirely from local lookups.

Core Claim

Each of the three directions supplies one piece, and each is weak alone. Web-scale n-gram validation is local: it judges whether a 2–7 word window is idiomatic and cannot see across sentences. Its best-known uses, such as Feature-Rich Statistical Translation of Noun Phrases and Entity Extraction with Knowledge from Web Scale Corpora, validate isolated phrases, and the former attributes 34% of its errors to words missing from the tables. Invented character names are the extreme case of such missing words, since the web has no counts for "Zarethin adjusted". Entity masking removes exactly the part of the window that makes lookups sparse, leaving a name-free context the web table can score.

Global narrative coherence supplies the aggregation. A single masked window proves nothing, but the distribution of web-rare contexts attached to one character across a 5,000-word story is a global object. It is the kind of long-range entity profile that Modeling Local Coherence: An Entity-Based Approach explicitly lacks, because its entity grid only records grammatical-role transitions between adjacent sentences.

The claim is therefore specific. Background-frequency weighting from a web corpus, applied to entity-masked contexts and aggregated per character across a story, detects character identity drift and cross-character contamination better than three alternatives:

  • the same signatures without web weighting;
  • lexicalized attribute tracking in the style of Re3;
  • similarity between passages.

It also provides a usable reranking signal for long-story generation.

The weighting is the frequency-ratio idea of Language Transfer Learning for Supervised Lexical Substitution, which compares a word's frequency in a specific context against a large background corpus. We move it from single-word substitution to entity profiles. Concretely, for character $e$ and masked context n-gram $c$, the signature weight is

$$w(e,c) = n_{\text{doc}}(e,c)\,\log\frac{N}{1+n_{\text{web}}(c)},$$

where $n_{\text{doc}}(e,c)$ counts occurrences of $c$ next to mentions of $e$ in the story segment, $n_{\text{web}}(c)$ is the web count of $c$, and $N$ is the web corpus size. The drift score between two segments $s$ and $t$ is one minus the cosine similarity between the weight vectors $w_s(e,\cdot)$ and $w_t(e,\cdot)$. We compare this with the same quantity computed against the story's other characters, which gives a contamination score.

Research Program

Signature extraction. A coreference resolver links names and pronouns to characters. Each mention is replaced by an empty slot, and we extract the left and right 2–4 token windows that do not include the slot, so every window is an ordinary name-free n-gram. Counts come from the Google N-gram / Google Books Ngram tables used in the prior work. Windows unseen on the web receive the maximum weight but are capped, so that noise does not dominate. Teaching Machines to Read and Comprehend warns that masking discards world knowledge. Here that is a feature for one reason and a cost for another. It is a feature because we want the profile to reflect what the story says rather than what a name evokes. It is a cost because masking inherits coreference errors, which is a documented fragility of masking, so we measure it directly by comparing gold coreference with system coreference on a subset.

Evaluation data. Detection is tested on controlled corruptions of real stories, following the logic of the permutation tests in entity-grid work. From WritingPrompts stories (the dataset of Hierarchical Neural Story Generation, averaging 734 words) and Wikiplots, we create three corruption types:

  • Identity splices: a passage about character A is replaced by a passage about a character from another story, with the name changed to A.
  • Within-story swaps: two characters' mentions are exchanged in one segment.
  • Explicit attribute flips: profession or age is changed.

Explicit attribute flips are the case Re3 is designed for, and we include them deliberately so the baseline has home ground.

Real errors are also needed. We generate stories of roughly 2,500 words with a Re3-style pipeline and collect human annotations of character-level inconsistency, separated into explicit-fact contradictions and softer voice or behavior drift. We also evaluate on the Contradiction Resolution questions of the BEAM benchmark from Beyond a Million Tokens that involve people. BEAM is conversational rather than narrative, so this is a transfer check, not the main test.

Use in generation. In the rewrite stage of Re3, candidate continuations are reranked on coherence and relevance. We add each candidate's drift and contamination scores against the running character signatures, and compare the resulting stories with Re3 using the same base model and the same number of candidates.

Baselines

Do-nothing reference. On detection this is chance, plus a length-matched score that flags segments by the number of new content words. For generation, it is Re3 with its original reranker and no added signal.

Lexicalized attribute tracking. This is the Attribute Dictionary edit module of Re3, which extracts explicit slots such as age and profession and flags conflicts. It is the strongest existing narrative-specific method for character contradictions. We run it with the same base model and coreference, on the same stories. We expect it to win on explicit attribute flips. Beating it means higher area under the ROC curve on splices, swaps, and the human-annotated soft-drift set, without losing more than a few points on attribute flips. Re3 reports a 14% absolute gain in stories judged to have a coherent overarching plot over rolling-window GPT-3. We found no result on Lacuna for its contradiction-detection accuracy as a standalone detector.

Entity grid. The model of Modeling Local Coherence reports 87.3% text-ordering accuracy against 72.1% for latent semantic analysis. We score each corrupted segment by its local transition likelihood. Because it abstracts entities to grammatical roles, a clean splice keeps every role transition intact, and we expect it to miss splices. Confirming that is part of the argument for a global, content-bearing profile.

Neural entity coherence. Entity-based Neural Local Coherence Modeling reports 76.54% on TOEFL essays and 77.52% on the NYT writing-quality task. We retrain it on our corruption training split under the same tuning budget.

Long-context model judge. An instruction-tuned model receives the whole story and is asked whether any character behaves inconsistently. Separately, it receives only the passages our method would compare.

Ablations. These isolate the key ingredient:

  • signatures with in-document weighting only, which removes the web weighting;
  • unmasked windows that include the name, which should collapse into sparsity for invented names;
  • masked windows weighted by a neural model's probability of the context instead of web counts.

Why not just…?

Why not just give the whole story to a long-context model? The main evidence against this is that Contradiction Resolution was the lowest-scoring category for every model on BEAM (Beyond a Million Tokens), and that full-history models degraded as sequences grew. Soft identity drift is subtler than a stated contradiction. Still, a strong judge shown only the right two passages may do very well. If the "judge on our selected passages" condition beats both our score and the whole-story judge, the honest conclusion is that signatures are best used to retrieve evidence, not to deliver verdicts. We would report the method as that retrieval step.

Why not just use a neural language model's probabilities as the background instead of web counts? This is the most serious threat, and the third ablation tests it head-to-head. The prior evidence only partly favors counts. Feature-Rich Statistical Translation of Noun Phrases found binary 7-gram existence beat continuous language models trained on the same web data. Adding 3-gram validation raised entity-matching AUC from 0.49 to 0.72 in Entity Extraction with Knowledge from Web Scale Corpora. Both results come from older models. If a modern model's surprisal matches web counts, the contribution narrows to the masked, per-character, web-weighted profile. The table then becomes an inexpensive deterministic implementation that makes no calls to a large model, and we would say so.

Why not just track embedding similarity between passages about a character? Modeling Local Coherence showed that overlap measures such as latent semantic analysis trail structured entity tracking. Entity-based Neural Local Coherence Modeling notes that low-quality text shows superficial similarity without coherent focus. Embeddings are dominated by generic scene content, which is exactly the part the web weighting discounts. We run sentence-embedding similarity on masked character passages as a baseline, and the no-web-weighting ablation shows how much of any gap comes from discounting common phrasing.

Proposed Experiments and Decisive Checks

The central test is detection AUC on identity splices and within-story swaps in WritingPrompts and Wikiplots. Web weighting must beat the no-weighting ablation, the entity grid, and Re3's attribute dictionary by a clear margin, with confidence intervals from bootstrapping over stories. If the unweighted signature performs the same, the web-scale component adds nothing and the proposal fails at its core.

The second test is the human-annotated soft-drift set from generated stories. Its value lies in showing gains where attribute dictionaries are blind by construction. Stratifying by story length (under 1,000 words, around 2,500, and above 5,000) tests whether the advantage grows with length, as a global profile should.

The third test is the neural-probability ablation, which decides whether the method should be presented as a detector or as a retrieval-plus-cost story.

Finally, a human pairwise evaluation compares Re3 with and without signature reranking, matched on base model and candidate count. It measures coherent plot, character consistency, and premise relevance. Success means a significant gain in character consistency with no loss in fluency or relevance.

A secondary check reports detection separately under gold and system coreference, and also by name type: invented names versus names common on the web. This confirms that masking is what makes the lookups work for invented characters.