SWAG: Storytelling With Action Guidance

Zeeshan Patel, Karim El-Refai, Jonathan Pei, Tianle Li

Abstract

Automated long-form story generation typically employs long-context large language models (LLMs) for one-shot creation, which can produce cohesive but not necessarily engaging content. We introduce Storytelling With Action Guidance (SWAG), a novel approach to storytelling with LLMs. Our approach reduces story writing to a search problem through a two-model feedback loop: one LLM generates story content, and another auxiliary LLM is used to choose the next best "action" to steer the story's future direction. Our results show that SWAG can substantially outperform previous end-to-end story generation techniques when evaluated by GPT-4 and through human evaluation, and our SWAG pipeline using only open-source models surpasses GPT-3.5-Turbo.

Introduction

While Large Language Models (LLMs) are remarkably good at writing short snippets, generating long-form stories that remain both coherent and captivating is a persistent challenge. Standard "end-to-end" (E2E) generation often results in stories that are technically sound but narratively "flat"—they lack the tension, pacing, and strategic plot developments that keep human readers engaged. SWAG (Storytelling With Action Guidance) transforms long-form story generation from a "one-shot" prompt into a strategic search problem where the AI must find the optimal path through a space of possible plot directions.

Methodology

The core of SWAG is a two-model feedback loop consisting of a Story Model ($\pi_{story}$), which writes the actual prose, and an Action Discriminator (AD LLM), which acts as a "director." At each step, the AD LLM chooses the best "action" (e.g., "add a plot twist," "add suspense," "add dialogue") from a predefined bank of 30 options to steer the story's direction.

The SWAG training and inference pipeline, involving preference data curation, supervised fine-tuning, and DPO alignment.

The training pipeline involves three key stages:

  1. Supervised Fine-Tuning (SFT): Base models like Llama-2-7B and Mistral-7B are fine-tuned on a dataset of 20,000 long stories to ensure they can handle long-context sequences (up to 32k tokens) using techniques like LongLoRA.
  2. Direct Preference Optimization (DPO): To train the "director," GPT-4 is used to generate preference data by ranking which action would be best to take next given a story state. The AD LLM is then aligned using DPO to mimic these high-quality strategic choices.
  3. The Inference Loop: The process is iterative: the Story Model writes a paragraph, the AD LLM picks an action, and the Story Model writes the next paragraph conditioned on that action.

Datasets

  • WritingPrompts: A standard dataset of human-written story prompts used as the starting point for all generations.
  • Synthetic Long Stories: 20,000 stories distilled from Llama-2, Mistral, and Mixtral for initial fine-tuning.
  • Action Preference Dataset: A newly curated dataset of ~60,000 prompts where GPT-4 ranked the "best next action" for various story states, used for the DPO alignment phase.

Key Insights: Overcoming the "Suspense" Bias

During data curation, researchers discovered that GPT-4 has a massive inherent bias toward choosing "add suspense." To prevent the AD LLM from becoming repetitive, the dataset was rebalanced by explicitly forcing the selection of other actions, resulting in much more diverse and interesting narratives.

Frequency of story actions in the rebalanced dataset, showing a diverse distribution of narrative elements like mystery, conflict, and suspense.

Key Results

The researchers evaluated SWAG using both GPT-4-as-a-judge and human evaluators across three metrics: Interest, Surprise, and Coherence.

  • Beating GPT-3.5-Turbo: A SWAG pipeline using only small, open-source models (like Mistral-7B) outperformed GPT-3.5-Turbo in human evaluations for "interesting-ness" and "surprise."
  • Win Rates: When judged by GPT-4, Mistral-7B with SWAG achieved a 68% win rate over the standard end-to-end Mistral-7B.
  • Human Preference: Humans overwhelmingly preferred SWAG-generated stories. For Llama-2-7B, the SWAG version was preferred over 80% of the time for interest and 90% for surprise, while coherence remained stable.

Conclusion and Future Work

SWAG demonstrates that modular, action-guided frameworks can significantly enhance the narrative quality of open-source LLMs. Future work may explore "test-time" action generation to move beyond a fixed action bank, as well as online reinforcement learning to further refine the director model's instincts. Additionally, the action-based interface offers a promising avenue for human-AI collaborative storytelling.

Summary updated

Abstract

Automated long-form story generation typically employs long-context large language models (LLMs) for one-shot creation, which can produce cohesive but not necessarily engaging content. We introduce Storytelling With Action Guidance (SWAG), a novel approach to storytelling with LLMs. Our approach reduces story writing to a search problem through a two-model feedback loop: one LLM generates story content, and another auxiliary LLM is used to choose the next best "action" to steer the story's future direction. Our results show that SWAG can substantially outperform previous end-to-end story generation techniques when evaluated by GPT-4 and through human evaluation, and our SWAG pipeline using only open-source models surpasses GPT-3.5-Turbo.
BibTeX
@inproceedings{patel2024swag,
  author = {Zeeshan Patel and Karim El-Refai and Jonathan Pei and Tianle Li},
  title = {{SWAG: Storytelling With Action Guidance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.824},
}
Figure