Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization

Adyasha Maharana, Mohit Bansal

Abstract

Adyasha Maharana, Mohit Bansal. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

Text-to-image synthesis has advanced rapidly, yet story visualization—generating a coherent sequence of images from a narrative—remains a significant challenge. Unlike single-image generation, story visualization requires maintaining narrative consistency across frames, parsing complex linguistic structures, and inferring implicit information not explicitly stated in the text. Researchers from UNC Chapel Hill address these hurdles with VLC-StoryGAN, a framework that integrates visuospatial, linguistic, and commonsense structures to produce high-quality, consistent narrative sequences.

Methodology

The VLC-StoryGAN architecture moves beyond "flat" text processing by incorporating three distinct structural priors to improve narrative flow and visual fidelity.

Memory-Augmented Recurrent Tree-Transformer (MARTT)

To handle linguistic complexity, the model utilizes constituency parse trees instead of flat word sequences. The MARTT component employs a bottom-up approach to understand how words compose into phrases. It uses memory cells to propagate state information between frames, ensuring that the narrative context is preserved as the story progresses.

Commonsense Graph Transformer

Captions are often sparse, omitting details like background elements. To bridge this gap, the model extracts entities from captions and retrieves related concepts from ConceptNet. These concepts are encoded via a Graph Transformer, providing the model with external visual context (e.g., associating "snow" with "white" and "cold") that is not explicitly mentioned in the text.

Visuospatial Dual Learning

The framework leverages a pre-trained dense captioning model to provide spatial feedback. During training, this model acts as a "spatial supervisor," checking if characters and objects in the generated images match the descriptions and are positioned correctly. This dual learning approach allows the model to learn spatial relationships without requiring manual bounding box annotations for the story dataset.

The VLC-StoryGAN architecture integrates MARTT for linguistic structure, a Graph Encoder for commonsense, and Dual Learning for visuospatial feedback.

Intra-Story Contrastive Loss

To improve the model's ability to distinguish between subtle narrative shifts (e.g., a character changing from "sighing" to "laughing"), a contrastive loss is applied. This forces the model to align specific words with corresponding image regions across the entire multi-frame sequence.

Evaluation

The model was evaluated on two primary benchmarks: PororoSV (cartoon-based, 10,000+ stories) and FlintstonesSV (CRAFT dataset).

Key Results

VLC-StoryGAN demonstrated substantial improvements over previous state-of-the-art models like StoryGAN and DuCo-StoryGAN:

  • Visual Quality: On PororoSV, the FID score improved significantly, dropping from 34.53 to 18.09.
  • Character Consistency: The character classification F1-score reached 43.02, indicating superior identity preservation across frames.
  • Human Preference: In head-to-head evaluations, humans preferred VLC-StoryGAN's output 62% of the time for visual quality.
Qualitative comparison between Ground Truth, DuCo-StoryGAN, and the proposed VLC-StoryGAN on the PororoSV dataset.

Discussion and Conclusion

The success of VLC-StoryGAN highlights that hierarchical linguistic structures are more effective than flat sequences for capturing complex character interactions. Furthermore, the use of "off-the-shelf" dense captioners demonstrates that spatial signals from real-world datasets can successfully transfer to cartoon domains.

However, challenges remain. The authors noted a slight dip in R-Precision, suggesting that while visual quality and consistency have improved, the global alignment between text and image still requires refinement. Additionally, the complexity of integrating three distinct structural priors necessitates careful hyperparameter tuning. Future work may focus on addressing the data scarcity and misalignment issues inherent in adapting video clips for static story visualization.

Summary updated

BibTeX
@inproceedings{maharana2021integrating,
  author = {Adyasha Maharana and Mohit Bansal},
  title = {{Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=TlCrwucTu2},
}
Figure