Introduction
While large language models like GPT-2 are capable of generating fluent text, they often struggle with long-form narrative generation. Specifically, these models frequently suffer from consistency issues—where plot points contradict earlier statements or character identities become muddled—and coherency issues, where the logical flow between sentences feels disjointed. This research proposes a two-stage framework designed to address these "marathon" challenges of automated storytelling.
Methodology: A Two-Stage Framework
The core of the proposed approach is a hierarchical generation process that separates high-level planning from low-level prose generation.
Stage 1: Outline Generation
Instead of generating a story from a prompt in a single pass, the model first produces an "outline." The authors experimented with both keywords and condensed abstracts, finding that abstracts (representing roughly 30% of the original text) provide a superior structural skeleton for the subsequent narrative.
Stage 2: Discourse-Aware Story Expansion
The second stage utilizes a Discourse-aware Transformer to expand the outline into a full story. To ensure logical flow and character consistency, the model incorporates two specialized auxiliary tasks:
- Discourse Coherency Enhancement: The model is trained to recognize discourse markers (e.g., and, but, because) between sentences, forcing it to learn the logical "connective tissue" of a narrative.
- Coreference Consistency Enhancement: By supervising the model's attention mechanism with known entity clusters (identified via Stanford’s CoreNLP), the model is encouraged to maintain accurate character tracking throughout the story.
Experimental Setup
The framework was trained and evaluated using the WritingPrompts dataset, which contains 300,000 stories from Reddit. For the discourse modeling task, the Book 8 Dataset (3.6 million sentence pairs) was used to train a BERT-based discourse marker predictor, providing "silver" labels for the story corpus.
Key Results and Analysis
The proposed model was benchmarked against standard GPT-2 and a pre-trained variant (GPT2P).
Quantitative Performance
As shown in the results table, the model significantly outperforms baselines in diversity (Distinct-1/2) and character tracking (Coreference Chains). Notably, when provided with ground truth outlines, the model achieves a dramatic reduction in perplexity, demonstrating the effectiveness of the two-stage approach.
Human Evaluation and Plot Control
In blind human evaluations, the model won decisively in Logicality, beating GPT2P in 31% of cases (with 47% ties). This indicates a substantial reduction in plot contradictions. Furthermore, visualization of attention weights confirmed a strong diagonal trend, proving that the expansion stage strictly adheres to the chronological order established in the generated outline.
Conclusion
This research demonstrates that hierarchical planning and auxiliary discourse/coreference tasks can significantly mitigate the common pitfalls of neural story generation. While a trade-off exists between strict plot control and raw fluency (perplexity), the gains in logical consistency represent a major step toward more reliable automated narrative systems. Future work will focus on improving prompt relevance and more sophisticated discourse parsing.
Summary updated