Abstract
Visual storytelling requires models to generate cohesive narratives from image sequences. However, standard models often fail to capture the contextual dependencies between steps, treating them as isolated events. We present V-Infill, a training technique that masks images within a sequence to force models to learn global narrative context. Alongside this method, we introduce ViPT (Visual Procedure Telling), a large-scale dataset of 46,000 visual procedures. Our results demonstrate that visual infilling significantly improves narrative coherence, state-of-the-art performance on procedural tasks, and robustness to missing visual data.
Introduction
While AI has achieved significant milestones in single-image captioning, the transition to visual storytelling—generating a cohesive sequence of sentences for a series of images—remains a formidable challenge. The primary obstacle is "contextual dependency": in a recipe or a multi-step craft, the description of a specific step depends heavily on the preceding and succeeding actions.
Traditional sequence-to-sequence models often struggle with this, leading to repetitive or contradictory narratives. Furthermore, real-world visual data is frequently incomplete or corrupted. While humans can easily infer a missing step in a visual sequence based on surrounding context, standard AI models often fail to bridge these information gaps. This paper addresses these issues through a novel training paradigm focused on visual infilling.
The V-Infill Framework
The core innovation of this work is Visual Infilling (V-Infill). The technique is designed to teach models to "read between the lines" by occasionally "blinding" the model during training. By replacing an image in a sequence with a zero tensor (masking), the model is forced to rely on the global narrative context to generate the correct textual description for that specific step.
Methodology
- Encoding: Visual features are extracted from each image using a ResNet-152 backbone.
- Narrative Context Layer: These features are passed through a bidirectional GRU. This allows the model to capture the flow of information both forwards and backwards through the sequence, ensuring that the representation of a masked step is informed by its neighbors.
- V-InfillR: A variant of the model employs a "scheduled" masking approach, where the number of masked images gradually increases during training to progressively challenge the model.
- Decoding: A standard GRU decoder generates the final text for each step using these context-aware features.
Datasets: ViST and ViPT
The researchers evaluated their approach using two primary datasets:
- ViST (Visual Storytelling): An established benchmark consisting of 50,000 creative stories.
- ViPT (Visual Procedure Telling): A new, massive dataset curated from "how-to" blogs, containing 46,200 procedures (approximately 340,000 image-text pairs) across categories such as recipes, crafts, and fitness.
Analysis of these datasets revealed that procedures (ViPT) have much stronger contextual ties than creative stories (ViST). In ViPT, only 39% of words in a given step are unique compared to the rest of the narrative, whereas in ViST, that figure is 60%. This higher level of overlapping context makes procedural data an ideal domain for infilling models to exploit.
Experimental Results
The V-Infill model demonstrated superior performance across several metrics and scenarios:
- State-of-the-Art Accuracy: On the ViPT dataset, the V-Infill model achieved a METEOR score of 27.51, significantly outperforming standard cross-entropy baselines.
- Human Evaluation: In head-to-head comparisons, human judges preferred the narratives generated by the V-Infill model 64% of the time over those from baseline models, citing better coherence and relevance.
- Robustness to Missing Data: When tested on sequences where an image was intentionally removed to simulate missing data, V-Infill maintained high BLEU and METEOR scores. In contrast, baseline models often resorted to generic or repetitive sentences when faced with a visual gap.
Discussion and Future Work
Despite its successes, the V-Infill approach faces certain limitations. The model occasionally produces "underspecified" sentences—generic phrases like "You will need the following ingredients"—when visual evidence is weak, rather than synthesizing specific details. Additionally, "copying errors" sometimes occur, where the model repeats phrases from adjacent steps instead of generating a unique description for the gap.
Future research will focus on "interpolating" between steps—generating descriptions for actions that have no corresponding image at all—and providing explicit guidance, such as keywords, to help the model fill in missing visual information with higher precision.
Summary updated