Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in short-form text generation, yet they frequently falter when tasked with long-form creative writing. A primary issue is pacing—the rate at which narrative information is revealed. LLMs often exhibit "unnatural pacing," where significant plot developments are compressed into single sentences while trivial details are expanded over multiple paragraphs. This paper, presented at EMNLP 2023 by Wang et al., introduces CONCOCT (CONCrete Outline ConTrol), a framework designed to regulate narrative flow through a specialized concreteness evaluator and a "vaguest-first" expansion strategy.
The Challenge of Hierarchical Planning
Standard hierarchical generation methods attempt to manage long-form content by expanding a high-level premise into a multi-level outline. However, these systems typically employ a "breadth-first" approach, expanding every branch of the story tree to a uniform depth. This structural rigidity ignores the semantic reality that some generated nodes are inherently more abstract or "vague" than others. The resulting narratives often feel "jarring" as the level of detail fluctuates inconsistently across different story segments.
The CONCOCT System
The authors propose moving away from structural depth toward semantic density. The CONCOCT system relies on four key components:
- Concreteness Evaluator ($M$): A RoBERTa-Large model trained to perform pairwise comparisons between text snippets to determine which is more "concrete" (possessing perceptible physical referents).
- Vaguest-First Expansion: Instead of expanding all nodes equally, CONCOCT identifies the most vague "leaf" node in the current outline and prioritizes it for expansion. This ensures that information gaps are addressed where they are most prominent.
- Concrete Children Generation: During expansion, the evaluator acts as a filter. New sub-events are only accepted if they are significantly more concrete than their parent node and sufficiently distinct from it.
- Concreteness Scheduler: To prevent repetitive detailing or "stalling," a scheduler increases the required "concreteness gap" as the outline grows.
Data and Training: GPT-BOOKSUM
A significant hurdle in training the concreteness evaluator was the presence of stylistic confounding variables. In the original BOOKSUM dataset, human-written chapter summaries often use bullet points, while paragraph summaries use prose. To prevent the model from simply learning to distinguish these styles, the authors created GPT-BOOKSUM. They utilized ChatGPT to rewrite summaries into a uniform prose style across all levels of granularity, ensuring the evaluator focuses strictly on information density.
Experimental Results
The system was evaluated using the WritingPrompts dataset. The specialized RoBERTa-based evaluator achieved 90% accuracy on the GPT-BOOKSUM test set, significantly outperforming zero-shot LLMs like GPT-4, which struggled to judge relative granularity.
In human pairwise evaluations, CONCOCT demonstrated a clear advantage in narrative control. Outlines generated by CONCOCT were preferred for their pacing over 60% of the time compared to breadth-first baselines. Furthermore, when these outlines were expanded into full stories, human readers preferred CONCOCT's pacing in 57% of cases.
Crucially, the focus on pacing did not degrade other narrative qualities. As shown in comparative testing, CONCOCT stories remained as coherent, relevant, and interesting as those produced by baseline models, suggesting that pacing is an additive quality that can be optimized independently.
Conclusion and Future Work
CONCOCT demonstrates that pacing is a semantic property best managed through targeted expansion rather than rigid tree structures. While the current system focuses on uniform pacing, future iterations could explore "pacing profiles" to allow for intentional dramatic variation—speeding up for action and slowing down for reflection. Currently, the system's reliance on human annotation for final evaluation highlights the ongoing need for more sophisticated automated metrics in creative writing tasks.
Summary updated