Language models writing multi-thousand-word stories fail in a characteristic way. Sentences stay fluent, facts mostly hold, and yet a character who was terrified three pages ago is suddenly calm and certain, with nothing in between to explain the change. Current long-story systems check facts that fit on a card, such as a character's age, job, or location. They do not check whether a character's inner life changes for a reason. Psycholinguistics already offers a cheap, interpretable way to measure that inner life: word lists that count how often text expresses anxiety, certainty, insight, causal reasoning, or a focus on the future. These counts have been used to track how real people's psychological states change over time. In story generation they have been applied only to five-sentence stories and coarse emotion labels.
We propose to measure each character's psycholinguistic state segment by segment across a long story. We then learn, from human-written fiction, how much that state normally changes given the events that happen in between. A change the events cannot explain gets a high "psychological surprise" score. We test two things. First, does this score find real coherence errors in long model-generated stories that fact checkers and generic coherence scorers miss? Second, does reranking candidate passages by the score make generated stories more coherent?
Core Claim
The claim is narrow and testable. Among coherence errors in long generated fiction, a meaningful share are unmotivated psychological shifts. By this we mean a character's emotion, certainty, or goal orientation reverses without a triggering event. These errors are not captured by the discrete attribute dictionaries of Re3 or the character and setting tracking of DOC. They are also not captured cleanly by dense-embedding coherence scorers. We predict that an event-conditioned surprise score over character-attributed psycholinguistic features will detect these errors better than three alternatives at matched cost: an unconditioned feature-jump score, a dense-embedding trajectory score, and a fine-tuned span detector. The key ingredient is that the score is conditioned on events. A large mood swing after a death is normal. The same swing after a character eats breakfast is not.
Relation to Prior Work
Two papers motivate the proposal by showing that psychological signals help narrative coherence, but only at micro-scale. Modeling Protagonist Emotions for Emotion-Aware Storytelling steers GPT-2 toward a target five-category emotion arc for a single protagonist. It uses reinforcement learning with a classifier reward, and human judges preferred its stories over 60% of the time against uncontrolled GPT-2. It works on 5-sentence ROCStories, and the authors themselves flag that sentence-level emotion inference ignores accumulated context. MLD-EA finds logic gaps in five-sentence Story Commonsense stories by tracking actions and Plutchik's eight basic emotions, and reports missing-sentence F1 of 76.04 against 22.14 for plain Llama3-8B. Our proposal differs from both in three ways. It works on stories of several thousand words. It uses continuous cognitive dimensions (certainty, tentativeness, insight, causation, temporal focus) alongside affect. And it asks whether a change was earned by intervening events, rather than whether a text matches a target arc or a gold emotion label.
A second group supplies the generation setting and constrains the evaluation. Re3 plans, drafts with distilled summaries, reranks candidate passages, and edits contradictions found through an attribute dictionary. DOC expands outlines into dense trees and adds a token-level discriminator. Both give us the pipeline into which our score is inserted, at Re3's rewrite stage. Neither tracks continuous psychological state.
SNaC constrains the design in two ways. It shows that span-level error categories raise annotator agreement (Krippendorff's α of 0.69 for character errors, versus 0.19 for holistic scales), and it shows that detectors trained on synthetic errors such as sentence shuffling fail on real model errors. We therefore evaluate on real generated errors and use synthetic perturbations only as a sanity check.
CohEval models stories as dense latent trajectories between fixed endpoints and trains a coherence evaluator on hard negatives. It is our closest methodological competitor, because it also scores trajectories. Its trajectories are uninterpretable, it scores whole short stories, and it cannot say which character broke or along which dimension.
Narrative Theory for Computational Narrative Understanding argues that computational work should model focalization, meaning whose perspective and consciousness a passage renders. We operationalize this by attributing text spans to characters before counting any features. PsyMatrix shows that hundreds of psycholinguistic features, summarized statistically, carry real signal about corpora. It aggregates statically over whole documents, whereas we treat the same kind of features as a time series.
Research Program
We build the system in three parts: a state extractor, a surprise model, and a reranker.
The state extractor splits a story into scenes of roughly 300 words. It runs coreference resolution and assigns each span to the characters it focalizes. A span belongs to a character if it is their dialogue, their reported thought, or narration whose subject is that character. This addresses the known weakness that dictionaries count words regardless of whose perspective they express. Over each character's spans in each scene, we compute a feature vector $z_{c,t}$ for character $c$ at scene $t$. The vector contains:
- LIWC-style category rates (affect, anxiety, certainty, tentativeness, insight, causation, past/present/future focus, first-person rate in dialogue), with negation scoping;
- NRC Affect Intensity scores;
- Valence-Arousal-Dominance coordinates.
The result is a vector of about 30 dimensions. If the proprietary LIWC dictionary is unavailable, open lexicons for the same categories substitute, and we report which were used.
The surprise model is a small predictor $f$, a fine-tuned encoder with a regression head. Its inputs are a character's recent state history $z_{c,<t}$ and a short text description $e_t$ of the events in scene $t$ that involve the character. We extract $e_t$ with an instruction-tuned model, deliberately stripped of emotion words so that it describes what happened, not how anyone felt. The model predicts the next state $z_{c,t}$ and a per-dimension variance $\sigma^2_{c,t}$. It is trained on human fiction: WritingPrompts stories and BOOKSUM chapters. The surprise for character $c$ at scene $t$ is the variance-normalized prediction error
$$S_{c,t} = \sum_{k} \frac{\left(z_{c,t,k} - \hat z_{c,t,k}\right)^2}{\sigma^2_{c,t,k}},$$
where $k$ indexes feature dimensions and $\hat z_{c,t}$ is the predicted state. The per-dimension terms tell a human editor which aspect of the character broke.
The reranker plugs $S$ into Re3's rewrite stage. Among candidate continuations, it penalizes those that produce high surprise for any active character. Candidates are otherwise scored exactly as Re3 already scores them.
Baselines
The naive reference has two parts. For detection, it flags each scene with probability equal to the base error rate. For generation, it runs the unmodified Re3 pipeline on the same premises with the same generator. For long-story coherence, the strongest reported gains in the file are DOC's 22.5% absolute improvement in plot coherence over Re3, and Re3's 14% absolute improvement over rolling-window generation (DOC, Re3).
We start from Re3 rather than DOC because Re3's rewrite stage is the natural insertion point. We also run DOC's outline expansion with and without our reranker, so the gain is measured on top of the stronger system. All generation arms use the same open generator, the same premises, and the same number of candidates per step. Only the reranking score differs.
The strongest detection competitor is a fine-tuned span detector in the style of SNaC. SNaC's fine-tuned T5 reports recall of 0.90 on character errors and 0.82 on scene-transition errors on real summary errors. We retrain it on our annotated training split with the same data and a matched hyperparameter search budget. For the new category of unmotivated psychological shift, we found no reported result on Lacuna, so the retrained detector's score on our test split is the bar. Beating it means higher AUROC and recall at matched precision on that category, with no loss on SNaC's existing categories when the two scores are combined.
A dense-trajectory competitor replaces our features with sentence embeddings of each character's spans. These are reduced by PCA to the same dimension and passed through an identically trained surprise model. We also run CohEval as a sequence scorer (CohEval). CohEval reports 67.6% on αNLI against 54.8% for RankGen-large. It has no reported result on long-story error detection on Lacuna.
A strong LLM judge is given the full story, or distilled summaries plus the current scene when the story exceeds its context. It is prompted with the exact error definitions used by annotators. Its cost is logged next to ours.
Three ablations isolate the proposal's parts:
- The unconditioned-jump ablation sets the event description $e_t$ to empty, which tests whether event conditioning matters.
- The no-attribution ablation computes features over the whole scene, which tests the focalization step.
- The static ablation uses PsyMatrix-style whole-story statistics, which tests whether trajectories matter at all.
Why not just…?
Why not just give a long-context model the whole story and ask it to find inconsistencies? This is the most serious threat. The file's evidence against it is indirect. Plain Llama3-8B reaches only 22.14 F1 on MLD-EA's gap detection. Re3 and DOC both found that raw long contexts produce drift, and SNaC found GPT-3 committing character and scene errors. None of this shows that a strong model prompted to judge will miss psychological shifts. We therefore run the LLM judge as a full baseline. If it matches our detector, the contribution shrinks to a cheaper, interpretable, per-dimension signal, and we will report it that way. We also test whether adding $S$ as an extra input to the judge improves its recall. That result would still justify the features.
Why not just add emotional state to Re3's attribute cards as text ("Mara: frightened")? Re3 already maintains character cards, so this is a cheap extension. We include it as a generation arm. After each scene, an LLM writes a one-line psychological summary per character into the card, and the edit stage checks for contradictions with it. This tests whether the continuous, event-conditioned score adds anything beyond a discrete label that the model itself maintains. We expect it to catch obvious reversals but to flag every justified change as well, because it has no model of what events license. The false-positive rate on human-written stories will settle this.
Why not just use dense embeddings, since the features are just a lossy projection? It is possible that sentence embeddings carry all the relevant signal and more. The matched dense-trajectory baseline tests exactly this. If dense trajectories detect equally well, the remaining argument for lexicon features is that they are auditable and diagnose which dimension broke. We would present that as the finding, rather than claiming a detection advantage.
Why not just train on synthetic errors? SNaC shows that shuffling-style errors do not transfer to real model errors, and CohEval shows that easy negatives teach surface matching. Our surprise model trains only on human stories, with no negatives at all. Real annotated errors are used only for evaluation and for the supervised baseline.
Proposed Experiments
Data. We generate 300 stories of 2,500 to 5,000 words from WritingPrompts premises, using Re3 and DOC-style pipelines with a shared open generator. Annotators then mark errors at the span level using SNaC's categories, plus one new category, "unmotivated psychological shift." We report Krippendorff's α per category; SNaC's 0.69 for character errors is the reference level. If agreement on the new category falls well below this, the category is too vague and we stop there.
First decisive check: detection. On a held-out split, we compare scene-level AUROC and recall at fixed precision for our score, every ablation, the dense-trajectory model, CohEval, the retrained SNaC-style detector, and the LLM judge. The proposal survives only if event-conditioned surprise beats the unconditioned-jump ablation and the dense-trajectory model on psychological-shift errors.
We also measure false positives on 100 held-out human-written stories from WritingPrompts and BOOKSUM chapters, which contain many earned shifts. The score must flag human stories far less often than generated ones.
Second decisive check: generation. We run blind pairwise human preference on coherence and premise relevance, the same axes Re3 and DOC report, comparing:
- Re3 with our reranker against plain Re3;
- Re3 with our reranker against Re3 with text state cards;
- DOC with our reranker against plain DOC.
The number of candidates per step is equal across arms. We also count annotated psychological-shift errors per 1,000 words in each arm.
Sanity check. On human stories, we swap a character's scene with the corresponding scene of a different character from the same story. This is used only to confirm that surprise rises, never as the headline result.
A clear negative result would also be informative. If surprise fails to separate earned from unearned shifts on human text, then these lexicon trajectories do not capture what makes a character's change believable.
References
- Re3: Generating Longer Stories With Recursive Reprompting and Revision
- DOC: Improving Long Story Coherence With Detailed Outline Control
- Modeling Protagonist Emotions for Emotion-Aware Storytelling
- MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions
- SNaC: Coherence Error Detection for Narrative Summarization
- Generating Contrastive Narratives Using the Brownian Bridge Process for Narrative Coherence Learning
- Narrative Theory for Computational Narrative Understanding
- Characterizing Text Datasets with Psycholinguistic Features