Abstract
Automated long-form story generation typically employs long-context large language models (LLMs) for one-shot creation, which can produce cohesive but not necessarily engaging content. We introduce Storytelling With Action Guidance (SWAG), a novel approach to storytelling with LLMs. Our approach reduces story writing to a search problem through a two-model feedback loop: one LLM generates story content, and another auxiliary LLM is used to choose the next best "action" to steer the story's future direction. Our results show that SWAG can substantially outperform previous end-to-end story generation techniques when evaluated by GPT-4 and through human evaluation, and our SWAG pipeline using only open-source models surpasses GPT-3.5-Turbo.
Introduction
While Large Language Models (LLMs) are remarkably good at writing short snippets, generating long-form stories that remain both coherent and captivating is a persistent challenge. Standard "end-to-end" (E2E) generation often results in stories that are technically sound but narratively "flat"—they lack the tension, pacing, and strategic plot developments that keep human readers engaged. SWAG (Storytelling With Action Guidance) transforms long-form story generation from a "one-shot" prompt into a strategic search problem where the AI must find the optimal path through a space of possible plot directions.
Methodology
The core of SWAG is a two-model feedback loop consisting of a Story Model ($\pi_{story}$), which writes the actual prose, and an Action Discriminator (AD LLM), which acts as a "director." At each step, the AD LLM chooses the best "action" (e.g., "add a plot twist," "add suspense," "add dialogue") from a predefined bank of 30 options to steer the story's direction.
The training pipeline involves three key stages:
- Supervised Fine-Tuning (SFT): Base models like Llama-2-7B and Mistral-7B are fine-tuned on a dataset of 20,000 long stories to ensure they can handle long-context sequences (up to 32k tokens) using techniques like LongLoRA.
- Direct Preference Optimization (DPO): To train the "director," GPT-4 is used to generate preference data by ranking which action would be best to take next given a story state. The AD LLM is then aligned using DPO to mimic these high-quality strategic choices.
- The Inference Loop: The process is iterative: the Story Model writes a paragraph, the AD LLM picks an action, and the Story Model writes the next paragraph conditioned on that action.
Datasets
- WritingPrompts: A standard dataset of human-written story prompts used as the starting point for all generations.
- Synthetic Long Stories: 20,000 stories distilled from Llama-2, Mistral, and Mixtral for initial fine-tuning.
- Action Preference Dataset: A newly curated dataset of ~60,000 prompts where GPT-4 ranked the "best next action" for various story states, used for the DPO alignment phase.
Key Insights: Overcoming the "Suspense" Bias
During data curation, researchers discovered that GPT-4 has a massive inherent bias toward choosing "add suspense." To prevent the AD LLM from becoming repetitive, the dataset was rebalanced by explicitly forcing the selection of other actions, resulting in much more diverse and interesting narratives.
Key Results
The researchers evaluated SWAG using both GPT-4-as-a-judge and human evaluators across three metrics: Interest, Surprise, and Coherence.
- Beating GPT-3.5-Turbo: A SWAG pipeline using only small, open-source models (like Mistral-7B) outperformed GPT-3.5-Turbo in human evaluations for "interesting-ness" and "surprise."
- Win Rates: When judged by GPT-4, Mistral-7B with SWAG achieved a 68% win rate over the standard end-to-end Mistral-7B.
- Human Preference: Humans overwhelmingly preferred SWAG-generated stories. For Llama-2-7B, the SWAG version was preferred over 80% of the time for interest and 90% for surprise, while coherence remained stable.
Conclusion and Future Work
SWAG demonstrates that modular, action-guided frameworks can significantly enhance the narrative quality of open-source LLMs. Future work may explore "test-time" action generation to move beyond a fixed action bank, as well as online reinforcement learning to further refine the director model's instincts. Additionally, the action-based interface offers a promising avenue for human-AI collaborative storytelling.
Summary updated