Translating Text Synopses to Video Storyboards

Xu Gu, Yuchong Sun, Feiyue Ni, Shizhe Chen, Ruihua Song, Boyuan Li, Xiang Cao

Introduction

In professional filmmaking, a storyboard serves as the essential bridge between a written script and the final screen production. It is a roadmap consisting of shot-by-shot images that visualize key plot points. While AI has made significant strides in text-to-video retrieval and story-to-image generation, these existing tasks often fail to capture the "cinematic flow" required for professional storyboarding.

Standard text-to-video models typically focus on short, redundant clips of a single action. Conversely, story-to-image tasks often rely on a rigid one-to-one mapping where every sentence describes exactly one image, leading to sequences that lack narrative transition. To address this gap, researchers have introduced TeViS (Text synopsis to Video Storyboard), a new task that requires models to translate abstract, high-level text synopses into cinematically coherent sequences of retrieved images.

The MovieNet-TeViS Dataset

To benchmark this task, the authors constructed the MovieNet-TeViS dataset, built upon the public MovieNet collection. This specialized dataset contains:

  • Scale: 10,000 text synopses paired with 45,584 manually selected keyframes.
  • Density: On average, each synopsis is paired with 4.6 keyframes.
  • Complexity: Unlike previous datasets such as LSMDC or VIST, MovieNet-TeViS features more abstract text and longer video segments (averaging 63.7 seconds).

This requires models to perform deep reasoning about plot progression and visual transitions—for example, moving from an establishing long shot of a house to a character close-up to reflect a shift in narrative focus.

Methodology: VQ-Trans

The researchers proposed VQ-Trans, a framework designed to reason through the visual flow of a story using three primary components:

  1. Joint Embedding Space: The model utilizes a fine-tuned CLIP (Contrastive Language-Image Pre-training) model to map both text and images into a shared mathematical space, ensuring that retrieved images are semantically relevant to the synopsis.
  2. Vector Quantization (VQ): To handle the massive visual diversity found in cinema, VQ-Trans does not process raw image features directly. Instead, a VQ module discretizes continuous visual features into a "codebook" of discrete tokens. This discretization acts as a denoiser, simplifying the visual space and making it easier for the model to learn the "language" of cinematography.
  3. Auto-regressive Decoder: The model employs a decoder-only Transformer architecture. It treats the text synopsis as a prefix and predicts visual tokens for the storyboard sequentially. This allows the model to consider the "history" of previous shots when choosing the next one, ensuring logical cinematic transitions.

Key Insights and Results

The study highlights that a successful storyboard is defined by cinematic coherence rather than simple relevance. A sequence must flow logically (e.g., establishing shot $\rightarrow$ medium shot $\rightarrow$ close-up). The authors found that treating text as a prefix in a decoder-only model was more effective for this task than traditional cross-attention mechanisms.

Performance was evaluated using Kendall’s $\tau$, a metric measuring sequence order accuracy (where 1.0 is a perfect match):

  • Ordering Task: When given the correct images in a shuffled order, VQ-Trans achieved a Kendall’s $\tau$ of 0.367, significantly outperforming CLIP-based baselines (0.244) and specialized story-to-image models like CNSI (0.182).
  • Retrieve-and-Order: In the more difficult task of selecting the correct images from a pool of 500 candidates and then ordering them, VQ-Trans maintained its lead with a score of 0.300.
  • The Human Gap: Despite these advancements, there remains a significant performance gap. Humans score 0.821 on the ordering task, suggesting that AI still has much to learn regarding the nuances of film direction and visual storytelling.

Conclusion and Future Work

TeViS establishes a new frontier for AI in the creative arts. While VQ-Trans demonstrates the power of vector quantization and auto-regressive reasoning, the authors acknowledge that future models must account for more complex cinematic elements such as specific camera angles, lighting styles, and movement. Furthermore, while this work focuses on retrieval, the VQ-Trans architecture provides a robust foundation for future generative models that could create entirely new storyboard images from scratch.

Summary updated

Abstract

A video storyboard is a roadmap for video creation which consists of shot-by-shot images to visualize key plots in a text synopsis. Creating video storyboards, however, remains challenging which not only requires cross-modal association between high-level texts and images but also demands long-term reasoning to make transitions smooth across shots. In this paper, we propose a new task called Text synopsis to Video Storyboard (TeViS) which aims to retrieve an ordered sequence of images as the video storyboard to visualize the text synopsis. We construct a MovieNet-TeViS dataset based on the public MovieNet dataset. It contains 10K text synopses each paired with keyframes manually selected from corresponding movies by considering both relevance and cinematic coherence. To benchmark the task, we present strong CLIP-based baselines and a novel VQ-Trans. VQ-Trans first encodes text synopsis and images into a joint embedding space and uses vector quantization (VQ) to improve the visual representation. Then, it auto-regressively generates a sequence of visual features for retrieval and ordering. Experimental results demonstrate that VQ-Trans significantly outperforms prior methods and the CLIP-based baselines. Nevertheless, there is still a large gap compared to human performance suggesting room for promising future work. The code and data are available at: \url{https://ruc-aimind.github.io/projects/TeViS/}
BibTeX
@inproceedings{gu2023translating,
  author = {Xu Gu and Yuchong Sun and Feiyue Ni and Shizhe Chen and Ruihua Song and Boyuan Li and Xiang Cao},
  title = {{Translating Text Synopses to Video Storyboards}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612417},
}
Figure