Abstract
Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transformation of these events into natural language sentences (event-to-sentence). However, typical neural language generation approaches to event-to-sentence can ignore the event details and produce grammatically-correct but semantically-unrelated sentences. We present an ensemble-based model that generates natural language guided by events. We provide results—including a human subjects study—for a full end-to-end automated story generation system showing that our method generates more coherent and plausible stories than baseline approaches.
Introduction
Automated story generation is a challenging task due to the massive space of possible narratives, which often leads to data sparsity when training neural networks directly on text. To mitigate this, researchers have adopted a two-stage pipeline: first generating a sequence of abstract plot points (events) and then "realizing" these events into natural language sentences. An event is typically represented as a 5-tuple: <subject, verb, object, preposition, modifier>.
While this decomposition simplifies the learning process, the second stage—event-to-sentence—presents its own difficulties. Standard sequence-to-sequence (seq2seq) models often act as simple language models, ignoring the input event if it contains novel combinations of tokens. This results in "hallucinated" text that is grammatically correct but semantically disconnected from the intended plot.
The Event-to-Sentence Ensemble
To address the fidelity-fluency tradeoff, we propose an ensemble-based cascading pipeline. This system attempts to realize an event using a series of models with varying levels of constraint, falling back to more flexible models if a confidence threshold is not met.
The cascade includes several specialized components:
- Retrieve-and-Edit (RetEdit): Finds the most similar event in the training data and uses a seq2seq model with attention to edit the associated training sentence. It is highly effective for complex structures but limited by the training distribution.
- Sentence Templating: Utilizing VerbNet and an AWD-LSTM, this model predicts syntactic structures and fills them with event tokens, ensuring semantic retention.
- Monte Carlo Beam Search: A modified beam search that optimizes for BLEU scores against the input event, promoting diversity.
- Finite State Machine (FSM) Constrained Beams: Forces the beam search through states representing the input tokens, guaranteeing their inclusion in the final sentence.
- Fallback Seq2Seq: A standard model used when other methods fail to meet confidence criteria.
Following generation, a Slot Filler replaces generalized tags (e.g., <PERSON>1) with specific entities using a dynamic memory graph to maintain narrative consistency across the story.
Experimental Evaluation
We evaluated our system using a custom Science Fiction TV Show Corpus comprising 2,276 story summaries from 11 long-running shows, including Doctor Who and The X-Files.
Quantitative Results
The full ensemble achieved a state-of-the-art Perplexity of 70.179, outperforming individual models like RetEdit (71.354) and FSM (104.775). Interestingly, the utilization of models shifted significantly between test sets and real-world generation. While RetEdit handled 82% of cases on the standard test set, its usage dropped to 31% when processing novel events from an AI plot generator, where Monte Carlo Beam Search became the primary driver (48%).
Human Subjects Study
We conducted a study on Amazon Mechanical Turk to compare our ensemble against a baseline seq2seq model across several qualitative metrics.
As shown in the evaluation results, the ensemble consistently outperformed the baseline. Participants rated the ensemble higher in grammar, quality, and adherence to the "Space Opera" genre. Notably, the improvements in genre adherence and overall enjoyability were statistically significant ($p < 0.10$).
Conclusion and Future Work
Our ensemble-based approach successfully balances the need for semantic fidelity with the desire for creative, fluent prose in automated story generation. However, challenges remain. The FSM constraints can occasionally lead to grammatical issues when forced to include many specific tokens. Additionally, the current slot-filling process relies on randomized selection from category lists. Future work will focus on integrating language models into the slot-filling stage to further enhance narrative coherence and word choice.
Summary updated