Improving Pacing in Long-Form Story Planning

Yichen Wang, Kevin Yang, Xiaoming Liu, Dan Klein

Abstract: Existing LLM-based systems for writing long-form stories or story outlines frequently suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details, resulting in a jarring experience for the reader. We propose a CONCrete Outline ConTrol (CONCOCT) system to improve pacing when automatically generating story outlines. We first train a concreteness evaluator to judge which of two events is more concrete (low-level-detailed). This evaluator can then be used to control pacing in hierarchical outline generation; in this work, we explore a vaguest-first expansion procedure that aims for uniform pacing. We further use the evaluator to filter new outline items based on predicted concreteness. Compared to a baseline hierarchical outline generator, humans judge CONCOCT's pacing to be more consistent over 57% of the time across multiple outline lengths; the gains also translate to downstream stories. All code, data, and models are open-sourced.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in maintaining local coherence, they often struggle with the "macro" level of storytelling—specifically pacing. In long-form narrative generation, pacing refers to the density of information and the level of detail provided over time.

Existing hierarchical planning systems typically expand story outlines using a "breadth-first" approach, treating every part of a story premise with equal weight. This often results in "unnatural pacing," where a high-stakes, complex journey might be summarized in a single sentence, while a minor incidental conversation is expanded into an entire chapter. This inconsistency creates a jarring experience for readers, where the narrative flow feels either rushed or unnecessarily bogged down. This paper introduces CONCOCT (CONCrete Outline ConTrol), a system designed to ensure a more consistent and natural flow in both outlines and the resulting full-length stories.

The CONCOCT System

The CONCOCT system introduces semantic control over the granularity of story events through three primary components: a specialized evaluator, a novel expansion heuristic, and a filtering mechanism.

Concreteness Evaluator ($M$)

At the core of the system is a RoBERTa-Large model trained to perform pairwise comparisons. Given two text snippets, the model determines which is more "concrete" (detailed and low-level) versus "vague" (abstract and high-level). Interestingly, the authors found that even state-of-the-art models like GPT-4 perform poorly at this specific task in zero-shot settings, necessitating a specialized, smaller model trained on targeted data.

Vaguest-First Expansion

Instead of expanding an outline tree uniformly (breadth-first), CONCOCT utilizes the evaluator to identify the "vaguest" leaf node in the current outline. This node is prioritized for expansion. By iteratively targeting the most abstract parts of the story, the system naturally balances the level of detail across the entire narrative, aiming for a uniform granularity that humans perceive as better pacing.

The CONCOCT expansion process: the evaluator identifies the vaguest leaf node (e.g., 'Lila causes chaos') for further expansion into more concrete sub-events.

Concreteness Filtering and Scheduling

During the expansion phase, the system generates multiple candidate sub-events using an LLM. The evaluator then filters these candidates to ensure that child nodes are significantly more concrete than their parent. To prevent the generation from stalling, a "concreteness scheduler" uses a threshold $T$ that decreases over time, balancing the requirement for detail against the risk of the LLM failing to produce sufficiently concrete expansions.

Data and Training

To train the evaluator, the authors introduced GPT-BOOKSUM, a dataset derived from the Project Gutenberg BOOKSUM corpus. They used ChatGPT to generate summaries at two distinct levels: Chapter-level (vague) and Paragraph-level (concrete). To ensure the model learned semantic granularity rather than surface-level features like length, they employed "dynamic pairing" with topic and length matching. This forced the model to distinguish between a long, vague summary and a short, concrete one.

Experimental Results

The system was evaluated by human annotators against a breadth-first baseline using premises from the WritingPrompts dataset. The results demonstrate that semantic control significantly improves perceived narrative quality.

Human evaluation results comparing CONCOCT to a baseline. CONCOCT shows significant improvements in pacing consistency and a reduction in nodes labeled as too vague or too detailed.

Key findings include:

  • Superior Pacing: Humans judged CONCOCT’s outlines to have more consistent pacing 61.5% of the time for short outlines and 65.0% for long outlines.
  • Error Reduction: The system significantly reduced the frequency of nodes marked as "too vague" or "too detailed" compared to the baseline.
  • Downstream Benefits: When outlines were expanded into full story excerpts, the pacing benefits persisted, with humans preferring CONCOCT-based stories 57.18% of the time.
  • Maintained Quality: Crucially, the improvement in pacing did not degrade other vital qualities such as plot coherence, relevance to the premise, or overall interestingness.

Conclusion and Future Work

CONCOCT demonstrates that pacing is a semantic property that can be effectively managed through specialized evaluation and non-uniform expansion strategies. While the current system aims for uniform pacing, professional storytelling often involves intentional pacing variations (e.g., accelerating through transitions or slowing down for climactic moments). Future research could explore using the concreteness evaluator to follow specific "pacing curves" or adapting the system to different domains and languages.

Summary updated

Abstract

Existing LLM-based systems for writing long-form stories or story outlines frequently suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details, resulting in a jarring experience for the reader. We propose a CONCrete Outline ConTrol (CONCOCT) system to improve pacing when automatically generating story outlines. We first train a concreteness evaluator to judge which of two events is more concrete (low-level-detailed). This evaluator can then be used to control pacing in hierarchical outline generation; in this work, we explore a vaguest-first expansion procedure that aims for uniform pacing. We further use the evaluator to filter new outline items based on predicted concreteness. Compared to a baseline hierarchical outline generator, humans judge CONCOCT's pacing to be more consistent over 57% of the time across multiple outline lengths; the gains also translate to downstream stories. All code, data, and models are open-sourced.
BibTeX
@inproceedings{wang2023improving,
  author = {Yichen Wang and Kevin Yang and Xiaoming Liu and Dan Klein},
  title = {{Improving Pacing in Long-Form Story Planning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.723},
}
Figure