Improving Pacing in Long-Form Story Planning

Yichen Wang, Kevin Yang, Xiaoming Liu, Dan Klein

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in short-form text generation, yet they frequently falter when tasked with long-form creative writing. A primary issue is pacing—the rate at which narrative information is revealed. LLMs often exhibit "unnatural pacing," where significant plot developments are compressed into single sentences while trivial details are expanded over multiple paragraphs. This paper, presented at EMNLP 2023 by Wang et al., introduces CONCOCT (CONCrete Outline ConTrol), a framework designed to regulate narrative flow through a specialized concreteness evaluator and a "vaguest-first" expansion strategy.

The Challenge of Hierarchical Planning

Standard hierarchical generation methods attempt to manage long-form content by expanding a high-level premise into a multi-level outline. However, these systems typically employ a "breadth-first" approach, expanding every branch of the story tree to a uniform depth. This structural rigidity ignores the semantic reality that some generated nodes are inherently more abstract or "vague" than others. The resulting narratives often feel "jarring" as the level of detail fluctuates inconsistently across different story segments.

The CONCOCT System

The authors propose moving away from structural depth toward semantic density. The CONCOCT system relies on four key components:

  1. Concreteness Evaluator ($M$): A RoBERTa-Large model trained to perform pairwise comparisons between text snippets to determine which is more "concrete" (possessing perceptible physical referents).
  2. Vaguest-First Expansion: Instead of expanding all nodes equally, CONCOCT identifies the most vague "leaf" node in the current outline and prioritizes it for expansion. This ensures that information gaps are addressed where they are most prominent.
  3. Concrete Children Generation: During expansion, the evaluator acts as a filter. New sub-events are only accepted if they are significantly more concrete than their parent node and sufficiently distinct from it.
  4. Concreteness Scheduler: To prevent repetitive detailing or "stalling," a scheduler increases the required "concreteness gap" as the outline grows.
The CONCOCT workflow: The system identifies the vaguest leaf node in an outline and expands it using a concreteness evaluator to filter generated sub-events.

Data and Training: GPT-BOOKSUM

A significant hurdle in training the concreteness evaluator was the presence of stylistic confounding variables. In the original BOOKSUM dataset, human-written chapter summaries often use bullet points, while paragraph summaries use prose. To prevent the model from simply learning to distinguish these styles, the authors created GPT-BOOKSUM. They utilized ChatGPT to rewrite summaries into a uniform prose style across all levels of granularity, ensuring the evaluator focuses strictly on information density.

Experimental Results

The system was evaluated using the WritingPrompts dataset. The specialized RoBERTa-based evaluator achieved 90% accuracy on the GPT-BOOKSUM test set, significantly outperforming zero-shot LLMs like GPT-4, which struggled to judge relative granularity.

In human pairwise evaluations, CONCOCT demonstrated a clear advantage in narrative control. Outlines generated by CONCOCT were preferred for their pacing over 60% of the time compared to breadth-first baselines. Furthermore, when these outlines were expanded into full stories, human readers preferred CONCOCT's pacing in 57% of cases.

Table comparing CONCOCT against a baseline across short and long outlines, highlighting the system's superior pacing and reduced instances of vagueness.

Crucially, the focus on pacing did not degrade other narrative qualities. As shown in comparative testing, CONCOCT stories remained as coherent, relevant, and interesting as those produced by baseline models, suggesting that pacing is an additive quality that can be optimized independently.

Conclusion and Future Work

CONCOCT demonstrates that pacing is a semantic property best managed through targeted expansion rather than rigid tree structures. While the current system focuses on uniform pacing, future iterations could explore "pacing profiles" to allow for intentional dramatic variation—speeding up for action and slowing down for reflection. Currently, the system's reliance on human annotation for final evaluation highlights the ongoing need for more sophisticated automated metrics in creative writing tasks.

Summary updated

Abstract

Existing LLM-based systems for writing long-form stories or story outlines frequently suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details, resulting in a jarring experience for the reader. We propose a CONCrete Outline ConTrol (CONCOCT) system to improve pacing when automatically generating story outlines. We first train a concreteness evaluator to judge which of two events is more concrete (low-level-detailed). This evaluator can then be used to control pacing in hierarchical outline generation; in this work, we explore a vaguest-first expansion procedure that aims for uniform pacing. We further use the evaluator to filter new outline items based on predicted concreteness. Compared to a baseline hierarchical outline generator, humans judge CONCOCT's pacing to be more consistent over 57% of the time across multiple outline lengths; the gains also translate to downstream stories. All code, data, and models are open-sourced.
BibTeX
@inproceedings{wang2023improving,
  author = {Yichen Wang and Kevin Yang and Xiaoming Liu and Dan Klein},
  title = {{Improving Pacing in Long-Form Story Planning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.723},
}
Figure