Reading Between the Lines: Exploring Infilling in Visual Narratives

Khyathi Raghavi Chandu, Ruo-Ping Dong, Alan W. Black

Abstract

Generating long-form narratives such as stories and procedures from multiple modalities has been a long-standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the surrounding contexts. The general seq2seq training methods render the models shorthanded while attempting to bridge the gap between these neighboring contexts. In this paper, we tackle this problem by using infilling techniques involving prediction of missing steps in a narrative while generating textual descriptions from a sequence of images. We also present a new large-scale visual procedure telling (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps using infilling technique demonstrates the effectiveness in visual procedures with more coherent texts. We conclusively show a METEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual storytelling. We also demonstrate the effects of interposing new text with missing images during inference.

Introduction: The Challenge of Visual Storytelling

While AI has become proficient at image captioning—describing a single photo—visual storytelling is a significantly more complex task. Humans do not merely describe what they see; they use surrounding context to infer what happened in the moments between images. Current sequence-to-sequence (seq2seq) models often struggle with this, treating steps in a sequence as isolated events and failing to capture the "subtext" that connects them. This frequently results in narratives that feel disjointed or repetitive.

To address this, researchers at Carnegie Mellon University investigated whether a model could be taught to bridge these information gaps by intentionally hiding parts of the sequence during training, a process known as infilling.

Methodology: Learning through Infilling

The proposed V-Infill technique moves beyond standard training by occasionally "masking" (hiding) an image during the training phase. This forces the model’s Narrative Context Layer to rely on the surrounding images to interpolate the missing local features and generate the correct textual description for the hidden step.

The V-Infill architecture utilizes a Narrative Context Layer to bridge gaps created by masked visual inputs.

The architecture includes:

  • Baseline (XE): A CNN-RNN architecture using ResNet-152 for visual features and a Bidirectional GRU to create a global narrative context.
  • V-Infill: The core innovation where an indicator function randomly samples indices to mask, forcing interpolation from global context.
  • V-InfillR: A variant using a "masking schedule" that gradually increases the difficulty by masking more features as training progresses.

The ViPT Dataset

The researchers introduced a massive new benchmark: ViPT (Visual Procedure Telling). It contains 46,200 procedures and approximately 340,000 image-text pairs across 10 domains, including recipes, crafts, and technology. Unlike existing creative storytelling datasets (like ViST), ViPT is goal-oriented. Procedures exhibit a much higher contextual overlap (61%) than stories (39%), making them an ideal test bed for infilling techniques where "Step B" is logically dependent on "Step A" and "Step C."

Key Results and Analysis

The V-Infill model achieved a METEOR score of 27.51 on procedures, surpassing previous state-of-the-art results in visual storytelling. Beyond quantitative metrics, the model showed significant qualitative improvements. When an image is missing during inference, baseline models often resort to "copying" text from the previous step. In contrast, the V-Infill model generates unique, contextually relevant text.

A qualitative comparison showing how V-Infill recovers from missing steps more effectively than the baseline XE model, which often repeats previous steps.

Key insights from the study include:

  • Implicit Knowledge: By simply masking inputs, the model implicitly learns the underlying structure of a task without needing external knowledge graphs.
  • Robustness: The model excels at "data imputation." If a user provides a sequence with a missing photo, V-Infill can still generate a coherent description of that missing step by analyzing the preceding and succeeding images.
  • Detail: V-Infill models produced longer, more detailed descriptions (average 76.49 words) compared to the baseline (71.26 words).

Limitations and Future Work

A persistent challenge remains under-specification, where models occasionally omit crucial details or generate generic instructions (e.g., "make sure it is properly cooked"). Future research may explore providing explicit guidance via keywords to control the content of infilled steps or even generating the missing images themselves to bridge visual gaps. These advancements could lead to more capable AI assistants for documenting complex real-world tasks.

Summary updated

Abstract

Generating long form narratives such as stories and procedures from multiple modalities has been a long standing dream for artificial intelligence. In this regard, there is often crucial subtext that is derived from the surrounding contexts. The general seq2seq training methods render the models shorthanded while attempting to bridge the gap between these neighbouring contexts. In this paper, we tackle this problem by using \textit{infilling} techniques involving prediction of missing steps in a narrative while generating textual descriptions from a sequence of images. We also present a new large scale \textit{visual procedure telling} (ViPT) dataset with a total of 46,200 procedures and around 340k pairwise images and textual descriptions that is rich in such contextual dependencies. Generating steps using infilling technique demonstrates the effectiveness in visual procedures with more coherent texts. We conclusively show a METEOR score of 27.51 on procedures which is higher than the state-of-the-art on visual storytelling. We also demonstrate the effects of interposing new text with missing images during inference. The code and the dataset will be publicly available at https://visual-narratives.github.io/Visual-Narratives/.
BibTeX
@inproceedings{chandu2020reading,
  author = {Khyathi Raghavi Chandu and Ruo-Ping Dong and Alan W. Black},
  title = {{Reading Between the Lines: Exploring Infilling in Visual Narratives}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.93},
}
Figure