Reading Between the Lines: Exploring Infilling in Visual Narratives

Khyathi Raghavi Chandu, Ruo-Ping Dong, Alan W. Black

Abstract

Visual storytelling requires models to generate cohesive narratives from image sequences. However, standard models often fail to capture the contextual dependencies between steps, treating them as isolated events. We present V-Infill, a training technique that masks images within a sequence to force models to learn global narrative context. Alongside this method, we introduce ViPT (Visual Procedure Telling), a large-scale dataset of 46,000 visual procedures. Our results demonstrate that visual infilling significantly improves narrative coherence, state-of-the-art performance on procedural tasks, and robustness to missing visual data.

Introduction

While AI has achieved significant milestones in single-image captioning, the transition to visual storytelling—generating a cohesive sequence of sentences for a series of images—remains a formidable challenge. The primary obstacle is "contextual dependency": in a recipe or a multi-step craft, the description of a specific step depends heavily on the preceding and succeeding actions.

Traditional sequence-to-sequence models often struggle with this, leading to repetitive or contradictory narratives. Furthermore, real-world visual data is frequently incomplete or corrupted. While humans can easily infer a missing step in a visual sequence based on surrounding context, standard AI models often fail to bridge these information gaps. This paper addresses these issues through a novel training paradigm focused on visual infilling.

The V-Infill Framework

The core innovation of this work is Visual Infilling (V-Infill). The technique is designed to teach models to "read between the lines" by occasionally "blinding" the model during training. By replacing an image in a sequence with a zero tensor (masking), the model is forced to rely on the global narrative context to generate the correct textual description for that specific step.

The V-Infill architecture, illustrating how a masked visual input is processed through a Narrative Context Layer to generate coherent text based on surrounding frames.

Methodology

  1. Encoding: Visual features are extracted from each image using a ResNet-152 backbone.
  2. Narrative Context Layer: These features are passed through a bidirectional GRU. This allows the model to capture the flow of information both forwards and backwards through the sequence, ensuring that the representation of a masked step is informed by its neighbors.
  3. V-InfillR: A variant of the model employs a "scheduled" masking approach, where the number of masked images gradually increases during training to progressively challenge the model.
  4. Decoding: A standard GRU decoder generates the final text for each step using these context-aware features.

Datasets: ViST and ViPT

The researchers evaluated their approach using two primary datasets:

  • ViST (Visual Storytelling): An established benchmark consisting of 50,000 creative stories.
  • ViPT (Visual Procedure Telling): A new, massive dataset curated from "how-to" blogs, containing 46,200 procedures (approximately 340,000 image-text pairs) across categories such as recipes, crafts, and fitness.

Analysis of these datasets revealed that procedures (ViPT) have much stronger contextual ties than creative stories (ViST). In ViPT, only 39% of words in a given step are unique compared to the rest of the narrative, whereas in ViST, that figure is 60%. This higher level of overlapping context makes procedural data an ideal domain for infilling models to exploit.

Experimental Results

The V-Infill model demonstrated superior performance across several metrics and scenarios:

  • State-of-the-Art Accuracy: On the ViPT dataset, the V-Infill model achieved a METEOR score of 27.51, significantly outperforming standard cross-entropy baselines.
  • Human Evaluation: In head-to-head comparisons, human judges preferred the narratives generated by the V-Infill model 64% of the time over those from baseline models, citing better coherence and relevance.
  • Robustness to Missing Data: When tested on sequences where an image was intentionally removed to simulate missing data, V-Infill maintained high BLEU and METEOR scores. In contrast, baseline models often resorted to generic or repetitive sentences when faced with a visual gap.

Discussion and Future Work

Despite its successes, the V-Infill approach faces certain limitations. The model occasionally produces "underspecified" sentences—generic phrases like "You will need the following ingredients"—when visual evidence is weak, rather than synthesizing specific details. Additionally, "copying errors" sometimes occur, where the model repeats phrases from adjacent steps instead of generating a unique description for the gap.

Future research will focus on "interpolating" between steps—generating descriptions for actions that have no corresponding image at all—and providing explicit guidance, such as keywords, to help the model fill in missing visual information with higher precision.

Summary updated

Abstract

Generating long form narratives such as stories and procedures from multiple modalities has been a long standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the surrounding contexts. The general seq2seq training methods render the models shorthanded while attempting to bridge the gap between these neighbouring contexts. In this paper, we tackle this problem by using \textit{infilling} techniques involving prediction of missing steps in a narrative while generating textual descriptions from a sequence of images. We also present a new large scale \textit{visual procedure telling} (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps using infilling technique demonstrates the effectiveness in visual procedures with more coherent texts. We conclusively show a METEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual storytelling. We also demonstrate the effects of interposing new text with missing images during inference. The code and the dataset will be publicly available at https://visual-narratives.github.io/Visual-Narratives/.
BibTeX
@inproceedings{chandu2020reading,
  author = {Khyathi Raghavi Chandu and Ruo-Ping Dong and Alan W. Black},
  title = {{Reading Between the Lines: Exploring Infilling in Visual Narratives}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.93},
}
Figure