Story Generation from Sequence of Independent Short Descriptions.

Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, Karthik SankaranarayananarXiv 2017 · OpenReview

TL;DR

This paper tackles the challenging task of generating coherent, human-like stories from a series of independent, short textual descriptions. The authors explore two main approaches: Statistical Machine Translation (SMT) and sequence-to-sequence deep learning models, evaluating their effectiveness on a publicly available dataset.

What Problem Does This Paper Address?

Current Natural Language Generation (NLG) systems often struggle with tasks requiring creativity, originality, and brevity, falling short of "strong AI" capabilities. This paper focuses on a specific aspect of this challenge: generating compelling narratives from fragmented descriptions of scenes or events. Unlike tasks where input is rich with context (like visual storytelling), this work aims to create stories from text alone, where the input descriptions, while informative, lack temporal context and a human touch. The goal is to bridge the gap between simple factual statements and engaging, cohesive narratives.

Methods

The authors investigate two primary paradigms for story generation:

  • Statistical Machine Translation (SMT):
    • Phrase-Based SMT (PB-SMT): This approach treats story generation as a translation problem, mapping phrases from the input descriptions to phrases in the target story. They utilize GIZA++ for alignment and the MOSES decoder for translation.
  • Syntax-Based SMT (SYNTAX-SMT): Similar to PB-SMT, but it focuses on translating syntactic units (like parse trees) rather than just word sequences, aiming for a more structured translation.
  • Deep Learning (Sequence-to-Sequence RNN):
    • They implement a Seq2Seq architecture using Recurrent Neural Networks (RNNs), specifically Gated Recurrent Units (GRUs).
  • A bidirectional encoder processes the input descriptions, and an attention-enabled decoder generates the story.
  • They experiment with different embedding dimensionalities for the RNNs (50, 128, and 256).

Datasets

The experiments were conducted using the "Text Annotations" from the recently introduced Visual Storytelling Dataset (VIST). Specifically, they utilized both "Descriptions of Images-in-Isolation (DII)" and "Stories of Images-in-Sequence (SIS)". The dataset consists of approximately 41,300 image sequences aligned with caption and story pairs. The text data was crowd-sourced, with participants generating descriptions and stories based on image sequences. The authors note that while the images were not used, the text pairs serve as a good proxy for their task, as both describe scenes or events.

Key Insights and Ideas

  • The Challenge of "Creativity": The paper highlights that generating truly creative and original stories is a significant hurdle for current NLG systems, even with advanced deep learning models.
  • Limitations of N-gram Metrics: Standard evaluation metrics like BLEU and ROUGE, commonly used for machine translation and summarization, may not adequately capture the nuances of story generation, potentially leading to lower scores even for qualitatively good outputs.
  • Seq2Seq vs. SMT Trade-offs: While SMT methods (particularly PB-SMT) achieved higher scores on n-gram based metrics, the Seq2Seq model showed better translation quality in terms of translation edit rate (TER), suggesting it might capture more coherent sentence structures.
  • The Need for Semantic Understanding: The models struggle to generate stories that are semantically related to the input descriptions beyond co-occurrence statistics, indicating a need for deeper semantic understanding.

Key Results

  • Quantitative Evaluation: Table 2 shows that SMT systems generally produced higher scores for BLEU, METEOR, and ROUGE-L, with PB-SMT performing best. The Seq2Seq models had lower scores on these metrics, likely due to their embedding-based nature which doesn't directly align with n-gram matching. However, the TER score for Seq2Seq was lower, suggesting better translation quality in terms of edit distance.
  • Qualitative Evaluation: Table 3 provides example case studies, demonstrating that all tested methods produced readable and coherent stories. However, the qualitative examples also underscore the difficulty in capturing the semantic essence and creativity expected in a good story.

Limitations and Future Work

  • Model Simplicity: The authors acknowledge that their current models are "too simplistic" to go beyond co-occurrence statistics and truly capture semantic relatedness.
  • Pretraining: A promising direction is to pretrain the encoder and decoder of the Seq2Seq model using large datasets of essays and novels to imbue them with broader linguistic knowledge.
  • Hierarchical RNNs: Exploring hierarchical recurrent neural networks is suggested as another avenue for improved story generation.
  • Beyond Translation: The paper emphasizes the need to move beyond pure translation-based techniques towards novel generation methods that can create stories semantically connected to the input, rather than just mapping words.
  • Trainable Metrics: Developing trainable metrics that can evaluate stories holistically, considering creativity, coherency, and novelty, is crucial for future progress.
  • Dataset Creation: Creating more appropriate datasets specifically for textual story generation is identified as an important future direction.

Summary updated

Abstract

Existing Natural Language Generation (NLG) systems are weak AI systems and exhibit limited capabilities when language generation tasks demand higher levels of creativity, originality and brevity. Effective solutions or, at least evaluations of modern NLG paradigms for such creative tasks have been elusive, unfortunately. This paper introduces and addresses the task of coherent story generation from independent descriptions, describing a scene or an event. Towards this, we explore along two popular text-generation paradigms -- (1) Statistical Machine Translation (SMT), posing story generation as a translation problem and (2) Deep Learning, posing story generation as a sequence to sequence learning problem. In SMT, we chose two popular methods such as phrase based SMT (PB-SMT) and syntax based SMT (SYNTAX-SMT) to `translate' the incoherent input text into stories. We then implement a deep recurrent neural network (RNN) architecture that encodes sequence of variable length input descriptions to corresponding latent representations and decodes them to produce well formed comprehensive story like summaries. The efficacy of the suggested approaches is demonstrated on a publicly available dataset with the help of popular machine translation and summarization evaluation metrics.
BibTeX
@misc{jain2017story,
  author = {Parag Jain and Priyanka Agrawal and Abhijit Mishra and Mohak Sukhwani and Anirban Laha and Karthik Sankaranarayanan},
  title = {{Story Generation from Sequence of Independent Short Descriptions.}},
  year = {2017},
  eprint = {1707.05501},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.05501},
}
Figure