Abstract
Visual storytelling is a task of creating a short story based on photo streams. Unlike existing visual captioning, storytelling aims to contain not only factual descriptions, but also human-like narration and semantics. However, the VIST dataset consists only of a small, fixed number of photos per story. Therefore, the main challenge of visual storytelling is to fill in the visual gap between photos with narrative and imaginative story. In this paper, we propose to explicitly learn to imagine a storyline that bridges the visual gap. During training, one or more photos is randomly omitted from the input stack, and we train the network to produce a full plausible story even with missing photo(s). Furthermore, we propose for visual storytelling a hide-and-tell model, which is designed to learn non-local relations across the photo streams and to refine and improve conventional RNN-based models. In experiments, we show that our scheme of hide-and-tell, and the network design are indeed effective at storytelling, and that our model outperforms previous state-of-the-art methods in automatic metrics. Finally, we qualitatively show the learned ability to interpolate storyline over visual gaps.
Introduction
While standard image captioning models excel at generating factual, literal descriptions of single images, they often struggle with the narrative flow required for visual storytelling. Visual storytelling involves taking a stream of photos—such as an album from a family trip—and generating a cohesive, human-like narrative.
The primary hurdle is the "visual gap": photo streams are temporally sparse, requiring an AI to infer what occurs between photos. This task demands imagination and subjective context rather than simple pixel translation. Existing models tend to overfit to visible images, resulting in disjointed and repetitive sentences. This paper introduces a novel training paradigm to bridge these gaps by explicitly teaching the model to "imagine" missing content.
Methodology: The Imagination Network
To address the visual gap, the authors propose a Hide-and-Tell learning scheme paired with an Imagination Network (INet).
Hide-and-Tell via Image Dropout
During training, the model is given a sequence of five photos, but one or two are randomly zero-masked (hidden). The network is penalized if it cannot produce the full, five-sentence ground-truth story. This forces the model to rely on surrounding context to hallucinate the missing narrative, acting as a powerful regularizer. To manage the difficulty of this task, the authors use Curriculum Learning, starting with all images visible and gradually increasing the number of hidden images as training progresses.
Architecture
The INet architecture follows a coarse-to-fine pipeline:
- Imagining Step: Uses a Bidirectional GRU combined with a 1D Non-Local (NL) self-attention layer to model long-range dependencies and reconstruct missing features.
- Telling Step: A second RNN-NL block refines these features to build a concrete semantic understanding of the whole sequence.
- Decoder: A standard GRU generates the final sentences word-by-word.
Experimental Setup
The model was evaluated on the VIST (Visual Storytelling) Dataset, the standard benchmark for this task. It contains 210,819 unique photos from 10,117 Flickr albums. Each data sample consists of five sequential images and five corresponding human-annotated narrative sentences.
Key Results and Insights
The proposed INet achieved new state-of-the-art results on the VIST dataset across all major automatic metrics, scoring 14.7 on BLEU-4, 35.6 on METEOR, and 10.0 on CIDEr. Ablation studies confirmed that the "hiding" step is crucial; removing it caused METEOR scores to drop from 35.6 to 35.1.
Story Interpolation
Because the model is explicitly trained to handle missing images, the authors introduced a novel task: Story Interpolation. By feeding the model images separated by blank frames, the model can successfully generate a dense narrative that smoothly interpolates the events happening in the gaps. Human evaluators preferred the Hide-and-Tell model 55.2% of the time compared to just 24.7% for the strongest baseline.
Conclusion
The Hide-and-Tell framework demonstrates that treating visual storytelling as an ill-posed reconstruction task forces models to perform narrative reasoning. While automatic metrics remain imperfect for creative text generation, the introduction of story interpolation opens exciting new directions for generating temporally dense narratives from sparse real-world photo albums.
Summary updated