Story Realization: Expanding Plot Events into Sentences

Prithviraj Ammanabrolu, Ethan Tien, Wesley Cheung, Zhaochen Luo, William Ma, Lara J. Martin, Mark O. Riedl

Abstract

Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transformation of these events into natural language sentences (event-to-sentence). However, typical neural language generation approaches to event-to-sentence can ignore the event details and produce grammatically-correct but semantically-unrelated sentences. We present an ensemble-based model that generates natural language guided by events. We provide results—including a human subjects study—for a full end-to-end automated story generation system showing that our method generates more coherent and plausible stories than baseline approaches.

Introduction

Automated story generation is a challenging task due to the massive space of possible narratives, which often leads to data sparsity when training neural networks directly on text. To mitigate this, researchers have adopted a two-stage pipeline: first generating a sequence of abstract plot points (events) and then "realizing" these events into natural language sentences. An event is typically represented as a 5-tuple: <subject, verb, object, preposition, modifier>.

While this decomposition simplifies the learning process, the second stage—event-to-sentence—presents its own difficulties. Standard sequence-to-sequence (seq2seq) models often act as simple language models, ignoring the input event if it contains novel combinations of tokens. This results in "hallucinated" text that is grammatically correct but semantically disconnected from the intended plot.

The Event-to-Sentence Ensemble

To address the fidelity-fluency tradeoff, we propose an ensemble-based cascading pipeline. This system attempts to realize an event using a series of models with varying levels of constraint, falling back to more flexible models if a confidence threshold is not met.

The end-to-end story generation pipeline, showing the transition from abstract events to realized sentences.

The cascade includes several specialized components:

  1. Retrieve-and-Edit (RetEdit): Finds the most similar event in the training data and uses a seq2seq model with attention to edit the associated training sentence. It is highly effective for complex structures but limited by the training distribution.
  2. Sentence Templating: Utilizing VerbNet and an AWD-LSTM, this model predicts syntactic structures and fills them with event tokens, ensuring semantic retention.
  3. Monte Carlo Beam Search: A modified beam search that optimizes for BLEU scores against the input event, promoting diversity.
  4. Finite State Machine (FSM) Constrained Beams: Forces the beam search through states representing the input tokens, guaranteeing their inclusion in the final sentence.
  5. Fallback Seq2Seq: A standard model used when other methods fail to meet confidence criteria.

Following generation, a Slot Filler replaces generalized tags (e.g., <PERSON>1) with specific entities using a dynamic memory graph to maintain narrative consistency across the story.

Experimental Evaluation

We evaluated our system using a custom Science Fiction TV Show Corpus comprising 2,276 story summaries from 11 long-running shows, including Doctor Who and The X-Files.

Quantitative Results

The full ensemble achieved a state-of-the-art Perplexity of 70.179, outperforming individual models like RetEdit (71.354) and FSM (104.775). Interestingly, the utilization of models shifted significantly between test sets and real-world generation. While RetEdit handled 82% of cases on the standard test set, its usage dropped to 31% when processing novel events from an AI plot generator, where Monte Carlo Beam Search became the primary driver (48%).

Human Subjects Study

We conducted a study on Amazon Mechanical Turk to compare our ensemble against a baseline seq2seq model across several qualitative metrics.

Human evaluation results comparing the ensemble model against a baseline across several qualitative metrics.

As shown in the evaluation results, the ensemble consistently outperformed the baseline. Participants rated the ensemble higher in grammar, quality, and adherence to the "Space Opera" genre. Notably, the improvements in genre adherence and overall enjoyability were statistically significant ($p < 0.10$).

Conclusion and Future Work

Our ensemble-based approach successfully balances the need for semantic fidelity with the desire for creative, fluent prose in automated story generation. However, challenges remain. The FSM constraints can occasionally lead to grammatical issues when forced to include many specific tokens. Additionally, the current slot-filling process relies on randomized selection from category lists. Future work will focus on integrating language models into the slot-filling stage to further enhance narrative coherence and word choice.

Summary updated

Abstract

Neural network based approaches to automated story plot generation attempt to learn how to generate novel plots from a corpus of natural language plot summaries. Prior work has shown that a semantic abstraction of sentences called events improves neural plot generation and and allows one to decompose the problem into: (1) the generation of a sequence of events (event-to-event) and (2) the transformation of these events into natural language sentences (event-to-sentence). However, typical neural language generation approaches to event-to-sentence can ignore the event details and produce grammatically-correct but semantically-unrelated sentences. We present an ensemble-based model that generates natural language guided by events.We provide results---including a human subjects study---for a full end-to-end automated story generation system showing that our method generates more coherent and plausible stories than baseline approaches.
BibTeX
@inproceedings{ammanabrolu2020story,
  author = {Prithviraj Ammanabrolu and Ethan Tien and Wesley Cheung and Zhaochen Luo and William Ma and Lara J. Martin and Mark O. Riedl},
  title = {{Story Realization: Expanding Plot Events into Sentences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=cGF5-Ymz7ho},
}
Figure