The Steep Road to Happily Ever after: an Analysis of Current Visual Storytelling Models

Yatri Modi, Natalie PardeNAACL 2019

TL;DR

This paper provides a comprehensive analysis of current visual storytelling models, identifying common error types and their frequencies. The authors offer concrete recommendations for improving future models, emphasizing the need for better sequential generation, temporal sequencing, and more robust evaluation metrics beyond automated scores.

What Problem Does This Paper Address?

Visual storytelling, the task of generating a cohesive, sequential narrative from a series of images, is a rapidly evolving area in AI. While image captioning focuses on describing individual images, visual storytelling requires understanding context, order, and inferring deeper meaning to create a compelling story. Existing models, though promising, still struggle with generating human-like narratives, often producing repetitive, nonsensical, or disconnected stories. This paper addresses the need for a systematic analysis of these models' shortcomings to guide future research and development in this complex domain.

Methods

The authors analyze three leading visual storytelling models: AREL (Adversarial Reward Learning), GLACNet (Glocal Attention Cascading Networks), and Contextualize, Show and Tell. Their analysis involves:

  • Model Cataloging and Comparison: Describing and contrasting the architectures and approaches of the selected models.
  • Performance Comparison: Evaluating models based on reported results and, where possible, re-implementing them to obtain consistent metrics.
  • Error Analysis: Systematically categorizing and quantifying common errors in generated stories. This involved defining specific error types like "Repetitions," "Description in Isolation," and "Ghost Entities."
  • Recommendations: Proposing concrete strategies to address identified limitations, drawing from their error analysis and insights from related work.

Datasets

The primary dataset used for training and evaluation in this work is the VIST Dataset.

  • VIST Dataset: This dataset consists of image sequences ("albums") paired with two types of captions: descriptions of images in isolation (DII) and stories of images in sequence (SIS). It contains over 10,000 Flickr albums with over 200,000 images, from which 50,200 story sequences were created by human annotators.

Key Insights and Ideas

  • Repetitions are a Major Hurdle: The most prevalent errors across models, particularly in AREL, are repetitions within stories and within sentences. This suggests a difficulty in maintaining narrative flow and avoiding redundancy.
  • "Description in Isolation" is a Common Pitfall: Many generated sub-stories sound like individual image captions rather than parts of a cohesive narrative. This indicates models may be defaulting to safer, less contextual descriptions.
  • Lexical Similarity, Semantic Redundancy: Models often generate semantically similar sentences that are lexically different, leading to redundancy. This might stem from the models' attempts to describe visually similar images in a sequence.
  • Creativity is Lacking: Stories often exhibit monotonous sentence structures and a lack of originality, suggesting a need for more sophisticated generation techniques.
  • Automated Metrics Fall Short: The paper validates previous findings that automated metrics like BLEU and CIDEr do not always correlate well with human judgments of story quality, highlighting the need for better evaluation methods.

Key Results

  • AREL vs. GLACNet: When re-implemented, AREL generally outperformed GLACNet across various metrics, including METEOR, CIDEr, ROUGE-L, and BLEU scores.
  • Error Frequencies: Repetitions and "Description in Isolation" were the most common error types observed. "Incomplete Stories" were the rarest, appearing only in AREL.
  • GLACNet's Strengths and Weaknesses: GLACNet showed fewer grammatical errors and repetitions due to its post-processing, but its stories were often monotonous.
  • Contextualize, Show and Tell: This model, which won the Visual Storytelling Challenge, also exhibited a high frequency of repetitions, despite generating sub-stories sequentially.

Limitations and Future Work

  • Limitations:
    • The analysis was limited to three models for which public code was available. Other models could not be fully analyzed.
  • The VIST dataset, while valuable, may contain human typing/grammar errors that affect training.
  • Automated metrics are not fully reliable for evaluating visual storytelling quality.
  • Future Work Recommendations:
    • Preprocessing Training Data: Automatically cleaning DII and SIS training files to reduce grammatical errors and improve model robustness.
  • Temporal Sequencing: Incorporating mechanisms to generate sub-stories sequentially rather than in parallel to reduce repetitions and improve coherence.
  • Feedback Mechanisms: Passing generated sub-stories or word tokens back as feedback to the model to maintain context and avoid inconsistencies.
  • Leveraging DII: Carefully using traditional image captions (DII) to aid in named entity recognition and reduce "ghost entities."
    • Visual Attention: Employing bottom-up and top-down visual attention mechanisms for better image grounding.
  • Machine Translation Approaches: Exploring neural machine translation models with visual encoders for more creative story generation.
  • Standardized Human Evaluation: Developing and using standardized human evaluation metrics to better assess story quality.

Summary updated

BibTeX
@inproceedings{modi2019steep,
  author = {Yatri Modi and Natalie Parde},
  title = {{The Steep Road to Happily Ever after: an Analysis of Current Visual Storytelling Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure