Abstract
Story visualization requires generating a sequence of images that maintain narrative coherence and character consistency. Traditional text-to-image models often fail to capture the hierarchical nature of language or the spatial relationships between objects. This paper introduces VLC-STORYGAN, a framework that integrates visuospatial, linguistic, and commonsense (VLC) structures to improve story generation. By employing a Memory-Augmented Recurrent Tree-Transformer (MARTT) for linguistic parsing, leveraging ConceptNet for external knowledge, and using a dense captioning model for spatial feedback, the proposed method significantly outperforms previous state-of-the-art models on the PororoSV and FlintstonesSV benchmarks.
Introduction
Generating a single image from a prompt is a well-studied task, but generating a story—a sequence of images where characters remain consistent and the narrative flows logically—presents unique challenges. Current models often treat input text as a "flat" bag of words, which leads to three primary failures in story visualization:
- Linguistic Complexity: Standard encoders miss the hierarchical structure of sentences (e.g., identifying who is performing an action on whom).
- Missing Context: Captions are often brief and lack the implicit commonsense knowledge (e.g., that "snow" implies a cold, white environment) necessary for visual grounding.
- Spatial Inconsistency: Models struggle to place characters and objects consistently across frames without explicit visuospatial feedback.
Methodology
VLC-STORYGAN addresses these gaps through a multi-pronged architecture designed to capture the underlying structure of the narrative.
Linguistic Structure (MARTT)
Instead of standard LSTMs, the model utilizes a Memory-Augmented Recurrent Tree-Transformer (MARTT). This component ingests constituency parse trees, allowing the model to understand the hierarchical relationship between words. By identifying the subject, verb, and object within a tree structure, the model can more accurately map linguistic roles to visual elements.
Commonsense Knowledge
To supplement brief captions, the model extracts key entities and retrieves related concepts from ConceptNet. These concepts are processed through a Graph Transformer, providing the generator with "hidden" details and context not explicitly mentioned in the text prompts.
Visuospatial Feedback and Contrastive Learning
The researchers implement a dual-learning approach using a pre-trained dense captioning model as a "critic." This critic, trained on real-world photos from the Visual Genome dataset, provides feedback on object placement and identification. Additionally, an Intra-Story Contrastive Loss is used to help the model distinguish between similar frames in a sequence by comparing word embeddings to specific image sub-regions, using adjacent frames as "negative" examples to sharpen the model's focus on subtle narrative changes.
Experiments and Results
The model was evaluated on two primary benchmarks: PororoSV, featuring characters from "Pororo the Little Penguin," and FlintstonesSV, adapted from the classic cartoon.
Quantitative Performance
VLC-STORYGAN demonstrated substantial improvements over previous models like StoryGAN and DuCo-StoryGAN:
- Image Quality: On the PororoSV dataset, the Fréchet Inception Distance (FID) improved from 34.53 to 18.09, indicating a significant jump in visual realism.
- Character Consistency: The Character F1 score rose from 38.01 to 43.02, showing better adherence to character identities across frames.
- Human Evaluation: In head-to-head comparisons, human judges preferred VLC-STORYGAN’s visual quality 62% of the time over the previous state-of-the-art.
Qualitative Analysis
Visual results indicate that VLC-STORYGAN produces clearer images with more accurate character placement and narrative relevance compared to baseline models.
Discussion and Conclusion
Ablation studies revealed that the linguistic tree structure (MARTT) provided the single largest boost in performance, underscoring the importance of grammar in visual grounding. Interestingly, the success of the dense captioning critic suggests that spatial relationships (like "sitting next to") are somewhat domain-independent, as a model trained on real photos effectively guided cartoon generation.
While the integration of commonsense knowledge provided modest gains, future work is needed to more deeply embed external knowledge graphs into the generation process. Overall, VLC-STORYGAN represents a major step forward in creating coherent, structurally-aware automated narratives.
Summary updated