Maintaining Global Coherence in Long Form Text Generation

1,334 concepts862 papers43 proposals

When a modern language model generates a single sentence, the result often feels flawless. The grammar is precise, the tone is natural, and the immediate logic holds. However, if you ask that same model to write a multi-chapter story, a long argumentative essay, or a comprehensive report, this illusion of intelligence quickly shatters. As the text grows, the model begins to wander, repeat itself, or flatly contradict what it wrote just a few paragraphs prior.

This breakdown highlights a fundamental challenge in natural language processing: the gap between local fluency and global narrative coherence. While models excel at predicting the next word, they struggle to sustain a unified logical thread over extended horizons.

The Root of the Problem: The Local-Global Gap

Why do language models lose their way over long distances? The issue stems from how autoregressive models are trained and optimized. They operate primarily on local statistical dependencies, maximizing the probability of the immediate next token given the preceding context.

As Ilya Sutskever et al. demonstrated in early character-level sequence generation work, models can master complex formatting and local syntactic correctness without possessing any inherent grasp of global semantic meaning. Decades later, even massively pretrained models suffer from this exact disconnect. Abigail See et al. observed that high syntactic fluency does not equate to long-term planning or world knowledge.

Without an explicit mechanism to anchor the generation process, a model's choices are dictated by immediate statistical probabilities. Over a long sequence, small local deviations accumulate. This leads to narrative drift, a failure mode where the text gradually loses alignment with its original premise or conditioning signal—a phenomenon also documented by Efstratios Gavves et al. in early conditional sequence models.

Anatomy of Narrative Drift

When global coherence fails, it manifests in several distinct ways across different text domains:

  • Internal Contradictions: Standard benchmarks often evaluate models on short-form next-token predictions, masking long-range instability. Domenic Rosati et al. showed that success in short-form metrics does not guarantee long-form consistency; models frequently introduce internal contradictions or revert to outdated knowledge when generating extended passages. In structured writing like argumentative essays, M. Xiao et al. noted a similar logical drift, where generated evidence inadvertently contradicts the primary assertion.
  • Loss of Causal and Entity Tracking: A cohesive narrative requires tracking abstract elements over long distances—such as keeping character traits stable or maintaining a timeline of events. Chao Zhao et al. highlighted that tracking these causal links and character relationships over extended horizons remains a primary bottleneck for automated systems, as models struggle to preserve the relative importance of a large cast of characters over time.
  • Wandering and Repetition: Without a high-level roadmap, models fall into repetitive loops or drift aimlessly from the plot. Haozhe Ji et al. identified this as a core failure mode occurring because models lack a high-level plan to bridge individual word selection with the overall document trajectory.

Bridging the Gap: Mechanisms for Global Control

To overcome the limitations of flat autoregressive generation, researchers have developed frameworks that introduce structural constraints, decoupling high-level planning from low-level token generation.

Hierarchical Planning and Blueprints

Instead of generating text word-by-word from scratch, hierarchical approaches force the model to outline a structural plan first. For example, Haozhe Ji et al. introduced DiscoDVT, a discourse-aware discrete variational transformer that learns to create a high-level structural blueprint of a story before filling in the details. This structural planning significantly reduces wandering and repetition.

Similarly, in the domain of persuasive writing, the PESA framework developed by M. Xiao et al. breaks down the writing process into a hierarchical tree of claims and evidence based on formal argumentation models, ensuring that the generated text remains anchored to the main thesis.

Flat Generation:
[Prompt] ──> [Token 1] ──> [Token 2] ──> [Token 3] ... ──> [Narrative Drift]

Hierarchical Generation:
[Prompt] ──> [Global Blueprint / Plot Outline]
                    │              │
                    ▼              ▼
             [Scene 1 Draft]   [Scene 2 Draft] ──> [Coherent Long Text]

Recursive Revision and Reprompting

Humans rarely write long-form text in a single, uninterrupted pass; we plan, draft, and revise. To mimic this process, Kevin Yang et al. developed the Re3 (Recursive Reprompting and Revision) framework.

Re3 uses distinct modules to handle different facets of the writing process. By recursively reprompting the model with the high-level plan and checking the generated text against the initial premise, it allows large language models to generate stories exceeding 2,000 words while maintaining plot coherence and relevance.

Explicit Intermediate Reasoning

Another strategy is to make the underlying logic or transitions explicit during the generation process. Debjit Paul et al. proposed the COINS framework, which completes missing parts of a story by explicitly generating intermediate commonsense inference rules. By separating the "reasoning" step from the "writing" step, the model maintains causal consistency across sentences.

In creative storytelling, this can also take the form of tracking abstract thematic constraints. Faeze Brahman et al. demonstrated that explicitly modeling a protagonist's emotional trajectory (e.g., joy → anger → sadness) allows models to adhere to a planned thematic sequence without sacrificing local fluency.

By moving away from pure next-token prediction and toward these multi-step, structured architectures, the field is transitioning from models that are merely fluent sentence-writers to systems capable of building sustained, coherent narratives.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 890 papers
  2. Version 1 · 2026-10-04 · 1077 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_global_narrative_coherence_3ad1d851,
  title = {global narrative coherence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-narrative-coherence/rdr_3ad1d851bda5409ea0934815708bdd78},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_global_narrative_coherence_3ad1d851_v2,
  title = {global narrative coherence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-narrative-coherence/rdr_3ad1d851bda5409ea0934815708bdd78/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake