Abstract: Existing LLM-based systems for writing long-form stories or story outlines frequently suffer from unnatural pacing, whether glossing over important events or over-elaborating on insignificant details, resulting in a jarring experience for the reader. We propose a CONCrete Outline ConTrol (CONCOCT) system to improve pacing when automatically generating story outlines. We first train a concreteness evaluator to judge which of two events is more concrete (low-level-detailed). This evaluator can then be used to control pacing in hierarchical outline generation; in this work, we explore a vaguest-first expansion procedure that aims for uniform pacing. We further use the evaluator to filter new outline items based on predicted concreteness. Compared to a baseline hierarchical outline generator, humans judge CONCOCT's pacing to be more consistent over 57% of the time across multiple outline lengths; the gains also translate to downstream stories. All code, data, and models are open-sourced.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in maintaining local coherence, they often struggle with the "macro" level of storytelling—specifically pacing. In long-form narrative generation, pacing refers to the density of information and the level of detail provided over time.
Existing hierarchical planning systems typically expand story outlines using a "breadth-first" approach, treating every part of a story premise with equal weight. This often results in "unnatural pacing," where a high-stakes, complex journey might be summarized in a single sentence, while a minor incidental conversation is expanded into an entire chapter. This inconsistency creates a jarring experience for readers, where the narrative flow feels either rushed or unnecessarily bogged down. This paper introduces CONCOCT (CONCrete Outline ConTrol), a system designed to ensure a more consistent and natural flow in both outlines and the resulting full-length stories.
The CONCOCT System
The CONCOCT system introduces semantic control over the granularity of story events through three primary components: a specialized evaluator, a novel expansion heuristic, and a filtering mechanism.
Concreteness Evaluator ($M$)
At the core of the system is a RoBERTa-Large model trained to perform pairwise comparisons. Given two text snippets, the model determines which is more "concrete" (detailed and low-level) versus "vague" (abstract and high-level). Interestingly, the authors found that even state-of-the-art models like GPT-4 perform poorly at this specific task in zero-shot settings, necessitating a specialized, smaller model trained on targeted data.
Vaguest-First Expansion
Instead of expanding an outline tree uniformly (breadth-first), CONCOCT utilizes the evaluator to identify the "vaguest" leaf node in the current outline. This node is prioritized for expansion. By iteratively targeting the most abstract parts of the story, the system naturally balances the level of detail across the entire narrative, aiming for a uniform granularity that humans perceive as better pacing.
Concreteness Filtering and Scheduling
During the expansion phase, the system generates multiple candidate sub-events using an LLM. The evaluator then filters these candidates to ensure that child nodes are significantly more concrete than their parent. To prevent the generation from stalling, a "concreteness scheduler" uses a threshold $T$ that decreases over time, balancing the requirement for detail against the risk of the LLM failing to produce sufficiently concrete expansions.
Data and Training
To train the evaluator, the authors introduced GPT-BOOKSUM, a dataset derived from the Project Gutenberg BOOKSUM corpus. They used ChatGPT to generate summaries at two distinct levels: Chapter-level (vague) and Paragraph-level (concrete). To ensure the model learned semantic granularity rather than surface-level features like length, they employed "dynamic pairing" with topic and length matching. This forced the model to distinguish between a long, vague summary and a short, concrete one.
Experimental Results
The system was evaluated by human annotators against a breadth-first baseline using premises from the WritingPrompts dataset. The results demonstrate that semantic control significantly improves perceived narrative quality.
Key findings include:
- Superior Pacing: Humans judged CONCOCT’s outlines to have more consistent pacing 61.5% of the time for short outlines and 65.0% for long outlines.
- Error Reduction: The system significantly reduced the frequency of nodes marked as "too vague" or "too detailed" compared to the baseline.
- Downstream Benefits: When outlines were expanded into full story excerpts, the pacing benefits persisted, with humans preferring CONCOCT-based stories 57.18% of the time.
- Maintained Quality: Crucially, the improvement in pacing did not degrade other vital qualities such as plot coherence, relevance to the premise, or overall interestingness.
Conclusion and Future Work
CONCOCT demonstrates that pacing is a semantic property that can be effectively managed through specialized evaluation and non-uniform expansion strategies. While the current system aims for uniform pacing, professional storytelling often involves intentional pacing variations (e.g., accelerating through transitions or slowing down for climactic moments). Future research could explore using the concreteness evaluator to follow specific "pacing curves" or adapting the system to different domains and languages.
Summary updated