Abstract
Visual storytelling is a task of creating a short story based on photo streams. Unlike existing visual captioning, storytelling aims to contain not only factual descriptions, but also human-like narration and semantics. However, the VIST dataset consists only of a small, fixed number of photos per story. Therefore, the main challenge of visual storytelling is to fill in the visual gap between photos with narrative and imaginative story. In this paper, we propose to explicitly learn to imagine a storyline that bridges the visual gap. During training, one or more photos is randomly omitted from the input stack, and we train the network to produce a full plausible story even with missing photo(s). Furthermore, we propose for visual storytelling a hide-and-tell model, which is designed to learn non-local relations across the photo streams and to refine and improve conventional RNN-based models. In experiments, we show that our scheme of hide-and-tell, and the network design are indeed effective at storytelling, and that our model outperforms previous state-of-the-art methods in automatic metrics. Finally, we qualitatively show the learned ability to interpolate storyline over visual gaps.
Introduction
Standard image captioning models are typically literal, focusing on describing the immediate contents of a single frame. In contrast, visual storytelling requires generating human-like, subjective, and narrative descriptions across a sequence of images. A significant hurdle in this domain is the visual gap. Because photo streams are temporally sparse—capturing only highlights rather than a continuous record—an AI must infer the connective narrative tissue between frames to tell a coherent story.
Existing models often overfit to the visible data in fixed-length datasets like VIST. To address this, we introduce a "hide-and-tell" training scheme. By randomly masking input photos, we force the model to learn contextual relations and deduce the missing storyline, effectively teaching it to "imagine" what happens between the frames.
The Hide-and-Tell Framework
To bridge visual gaps, we propose the Imagination Network (INet), which utilizes a novel learning scheme and architecture designed for relational reasoning.
Hide-and-Tell Training and Curriculum Learning
During training, the model receives a sequence of five photos, but one or two are randomly zero-masked (hidden). The network is penalized if it fails to generate the full five-sentence story, forcing it to use surrounding context to reconstruct the missing narrative. To manage the difficulty of this task, we employ curriculum learning: training begins with zero hidden images and gradually increases the masking frequency as the model's performance saturates.
Imagination Network Architecture
The INet architecture follows a coarse-to-fine pipeline consisting of three primary stages:
- Imagining Step: A Bidirectional GRU combined with a 1D Non-Local (NL) self-attention block analyzes the sequence of image features. It captures inter-frame relations to roughly reconstruct the features of hidden images.
- Telling Step: A second RNN-NL block refines these reconstructed features to establish concrete semantics across the entire photo stream.
- Decoder: A standard GRU generates the final sentences based on these refined features.
Experimental Results
We evaluated our model on the VIST (Visual Storytelling) Dataset, which contains over 210,000 unique photos from Flickr albums.
Quantitative Performance
The proposed INet achieves new state-of-the-art results across standard automatic metrics, including BLEU-4 (14.7), METEOR (35.6), ROUGE-L (29.7), and CIDEr (10.0). Ablation studies confirmed that both the "hide-and-tell" masking and the non-local attention layers are critical to these gains.
Qualitative and Human Evaluation
Because automatic metrics are often insufficient for narrative tasks, we conducted a human study. Judges preferred the Hide-and-Tell model 55.2% of the time, compared to 24.7% for a strong baseline. Qualitative analysis shows that the model effectively borrows context from distant photos to fill in narrative blanks.
Conclusion
The Hide-and-Tell scheme demonstrates that masking input frames acts as more than a regularizer; it actively encourages the network to learn narrative flow. By treating images as relational tokens, the model can interpolate storylines over visual gaps. Future work may extend this to "Story Interpolation," where models generate dense, detailed narratives from even sparser photo sets, creating a more expressive bridge between visual data and human language.
Summary updated