TL;DR
Neural story generation often produces locally fluent but globally incoherent text. This paper introduces a novel content planning approach that leverages Aristotle's principles of good storytelling (event choice, character, relevance, diction) to guide plot generation, significantly improving story quality and relevance.
What Problem Does This Paper Address?
Large language models can generate text that sounds human-like, but when it comes to creative writing, especially long-form stories, they struggle with structure and long-range coherence. This leads to stories that meander, lack a clear narrative arc, and often fail to stay on topic, even when given a specific prompt. The authors argue that this problem stems from a lack of principled content planning, which is crucial for building a good story.
Methods
The core of this work is a two-stage generation process:
- Content Planning (Plot Generation): Instead of directly generating a story, the system first generates a plot. This plot is represented using Semantic Role Labeling (SRL) tags. The novelty here lies in using Aristotelian rescoring models to refine this plot. These rescorers are trained as classifiers to identify "good" plot structures based on Aristotle's principles:
- Event Choice and Arrangement: Models ensure logical event sequencing and coherence.
- Character: Models track character presence and actions.
- Relevant Content: Models ensure the plot stays on topic with the prompt.
-
Diction: (Implicitly handled by the downstream language model). These rescorers act as "judges" for the generated plot, providing feedback that guides the plot generation model. The authors use a revision-based approach, where an initial plot is generated and then iteratively refined by these rescorers.
-
Surface Realization (Story Generation): A pre-trained language model (BART) then takes the refined plot and the original prompt to generate the final story. This leverages BART's strength in producing fluent and grammatically correct text, now guided by a structured plot.
Datasets
- Writing Prompts Dataset: This is the primary dataset used for training and evaluation. It consists of user-generated stories paired with their prompts. The authors note that this dataset has a "one-to-many" relationship between prompts and stories and contains some prompt artifacts. They also mention filtering it to reduce lexical overlap between training and test prompts to better simulate real-world performance.
Key Insights and Ideas
- Aristotle's Poetics as a Blueprint for AI Storytelling: The paper demonstrates that ancient literary principles can be effectively translated into computational models for guiding creative AI. This suggests a rich avenue for incorporating qualitative aspects of storytelling into NLP.
- Rescoring for Principled Plot Generation: Using specialized rescoring models (trained on positive and negative examples of plot structures) is a powerful way to inject desired narrative qualities into the plot generation process, going beyond simple statistical patterns.
- The Importance of Plot Structure: The results strongly suggest that a well-formed plot is a prerequisite for generating high-quality, coherent, and relevant stories, even with powerful pre-trained language models.
- Human Evaluation is Crucial: While automatic metrics provide some signal, human evaluation revealed significant differences that automatic metrics missed, highlighting the complexity of assessing story quality.
Key Results
- Improved Story Quality and Relevance: The proposed Aristotelian Plot system significantly outperformed baselines (Naive Plot, Prompt-to-Story, and two other state-of-the-art systems) in both automatic and human evaluations.
- Human Evaluation Wins: Pairwise human comparisons showed the Aristotelian Plot system was superior to both the Prompt-to-Story and Naive Plot baselines in terms of both Relevance and Overall Quality.
- Automatic Metrics: The system showed improvements in metrics like Vocab:Token Ratio and Relevance, though a gap still exists between generated and gold plots.
- Rescorer Performance: RoBERTa-large models achieved the highest accuracy across various Aristotelian rescoring tasks, indicating their effectiveness in capturing narrative principles. The ensemble of all rescorers performed best.
Limitations and Future Work
- Plot Representation Gap: While the Aristotelian plot improved generation, there's still a gap between generated and "gold" plot structures.
- Story Model Incorporation: The authors hypothesize that further improvements could be made by enhancing the story generation model's ability to better incorporate the generated plot.
- Complex Prompts: The system struggled with highly abstract or meta-level prompts (e.g., "Write a story where I can't tell whether the protagonist is a hero or a villain"), indicating a need for finer control over plot and story structure.
- Future Work: The authors suggest exploring interactions between rescorers, investigating Pareto fronts for rescorer combinations, and developing models that can better incorporate plots. They also note the difficulty of human evaluation for long stories and funding constraints.
Summary updated