Content Planning for Neural Story Generation with Aristotelian Rescoring

Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph M. Weischedel, Nanyun Peng

TL;DR

Neural story generation often produces fluent but incoherent narratives. This paper introduces a novel content planning approach that leverages Aristotle's principles of good storytelling (event choice, character, relevance) to guide plot generation. By using rescoring models trained on these principles, the system generates more relevant and higher-quality stories compared to existing methods.

What Problem Does This Paper Address?

Large language models (LLMs) can generate human-like text, but struggle with long-form narrative generation. Specifically, generated stories often lack global structure and cohesion, veering between topics and failing to maintain relevance to the initial prompt. This results in stories that feel disjointed and lack a clear narrative arc, even if individual sentences are fluent. The authors aim to address this by improving the "content planning" stage of story generation, focusing on creating a coherent and principled plot structure before the final text is generated.

Methods

The core of the proposed system is a two-stage process:

  1. Plot Generation: A BART language model is fine-tuned to generate a plot structure from a given prompt. This plot is represented using Semantic Role Labeling (SRL) tags.
  2. Aristotelian Rescoring: Instead of directly generating a story from the initial plot, the system uses an ensemble of "rescoring models." These models are trained to evaluate and improve the generated plot based on principles derived from Aristotle's Poetics:
  3. Event Choice and Arrangement: Models are trained to distinguish good event sequences from bad ones (e.g., "grow -> began -> fade" is good, "ended -> lived -> born" is bad). This is achieved through various "shuffling" techniques to create negative examples.
  4. Character: Models learn to predict when characters should appear or reappear in the plot.
  5. Relevant Content: Models are trained to assess the relevance of a plot to the original prompt, distinguishing between relevant and randomly generated plots.
  6. Diction: This aspect is not explicitly modeled by a rescorer, as it's considered less critical for plot structure.

The rescoring models provide scores that are used to re-rank and refine the initially generated plot. This improved plot then serves as input for a second BART model to generate the final story. The system uses a modified decoding objective that incorporates the scores from these Aristotelian rescorers.

Datasets

  • Writing Prompts Dataset: This dataset, consisting of user-generated stories and their associated prompts from Reddit, is used for both training and evaluation. The authors note its suitability due to its large scale and diverse long-form stories.
  • Silver-Standard Plots: Plots are extracted from the Writing Prompts dataset using SRL and coreference resolution. These are referred to as "silver-standard" plots, meaning they are derived from existing stories rather than being human-authored gold plots.

Key Insights and Ideas

  • Aristotle's Poetics as a Framework: The paper successfully demonstrates that classical literary principles can be operationalized as rescoring models to guide neural generation. This bridges the gap between literary theory and computational creativity.
  • The Power of Rescoring for Plot Refinement: Instead of relying on a single plot generation model, using multiple rescorers, each focusing on a specific storytelling aspect, leads to a more robust and principled plot. The ensemble approach proved more effective than individual rescorers.
  • Plot Structure is Crucial for Cohesion: The results strongly suggest that a well-structured plot, guided by principles of good storytelling, is essential for generating coherent and relevant long-form narratives, even with powerful LLMs.
  • Human Evaluation is Key: While automatic metrics show improvements, human evaluation reveals the true impact on story quality, highlighting the limitations of purely lexical or structural metrics for complex tasks like story generation.

Key Results

  • Improved Relevance and Quality: Stories generated using the Aristotelian rescoring approach showed significant improvements in relevance and overall quality compared to baselines that either did not use content planning or used an unprincipled approach.
  • ROBERTa Dominance: RoBERTa-large models achieved the highest accuracy across all Aristotelian rescoring tasks, outperforming XGBoost and CNN-based models.
  • Inter-sentence Shuffling is Hardest: The inter-sentence shuffled event rescoring task was the most difficult, reflected in lower accuracies for baseline models.
  • Ensemble Wins: The ensemble of all five rescorers, with learned mixture weights, achieved the best performance in terms of ranking accuracy.
  • Human Evaluation Wins: Pairwise human comparisons showed the Aristotelian Plot system significantly outperforming both the "Prompt to Story" (direct generation) and "Naive Plot" (plot without rescoring) baselines in both Relevance and Overall Quality.

Limitations and Future Work

  • Gap with Gold Plots: While the Aristotelian plot improved over naive plots, a significant gap remains between generated and gold plot structures.
  • Story Model Improvements: The authors hypothesize that further improvements could be achieved by modifying the story generation model to better incorporate the refined plot structure.
  • Investigating Rescorer Interactions: Further work is suggested to analyze the interactions between different rescorers and explore Pareto fronts for optimal combinations.
  • Handling Complex Prompts: The system struggled with highly abstract or meta-level prompts (e.g., "Write a story where I can't tell whether the protagonist is a hero or a villain"), indicating that finer control over plot and story structure is needed for such cases.
  • Human Evaluation Cost: The authors note the difficulty and cost of human evaluation, which limited the scope of their experiments.

Summary updated

Abstract

Long-form narrative text generated from large language models manages a fluent impersonation of human writing, but only at the local sentence level, and lacks structure or global cohesion. We posit that many of the problems of story generation can be addressed via high-quality content planning, and present a system that focuses on how to learn good plot structures to guide story generation. We utilize a plot-generation language model along with an ensemble of rescoring models that each implement an aspect of good story-writing as detailed in Aristotle's Poetics. We find that stories written with our more principled plot-structure are both more relevant to a given prompt and higher quality than baselines that do not content plan, or that plan in an unprincipled way.
BibTeX
@inproceedings{goldfarbtarrant2020content,
  author = {Seraphina Goldfarb-Tarrant and Tuhin Chakrabarty and Ralph M. Weischedel and Nanyun Peng},
  title = {{Content Planning for Neural Story Generation with Aristotelian Rescoring}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.351},
}
Figure