TeViS:Translating Text Synopses to Video Storyboards

Xu Gu, Yuchong Sun, Feiyue Ni, Shizhe Chen, Xihua Wang, Ruihua Song, Boyuan Li, Xiang Cao

TL;DR

The paper introduces TeViS (Text synopsis to Video Storyboard), a novel multimodal task aimed at translating abstract text synopses into cinematically coherent sequences of keyframes. To benchmark this task, the authors release the MovieNet-TeViS dataset and propose VQ-Trans, an auto-regressive decoder-only transformer that leverages vector quantization to significantly outperform traditional cross-modal retrieval methods.

What Problem Does This Paper Address?

When directors plan a film, they don't just look for images that match a script; they design a storyboard—a sequence of shots that flow together cinematically (e.g., transitioning from an establishing long shot to a dramatic close-up).

Existing text-to-vision methods fall short here. Text-to-video retrieval focuses on short, isolated clips with highly redundant frames. Meanwhile, story-to-image retrieval relies on a strict one-to-one mapping from detailed sentences to individual images, completely ignoring the fluid "lens language" required for video creation.

Comparison of different cross-modal retrieval tasks.

The TeViS task challenges models to take a brief, high-level text synopsis and retrieve an ordered, cinematically smooth sequence of images from a massive movie database, effectively acting as an automated assistant for amateur video creators.

Methods

The authors propose VQ-Trans, a framework built on a decoder-only transformer architecture designed to handle both image ordering and variable-length sequence generation.

  1. Text Prefix Encoding: The input text synopsis is encoded using a fine-tuned CLIP text encoder. These text features serve as a "prefix" hint for the transformer decoder.
  2. Image Discretization via Vector Quantization (VQ): Learning the language of cinematography from continuous, highly diverse image features is notoriously difficult. To stabilize training, VQ-Trans uses a fine-tuned CLIP image encoder combined with a VQ codebook ($C$) to quantize continuous visual features into discrete visual tokens.
  3. Auto-regressive Generation: The transformer decoder ingests the text prefix and the history of previous discrete image tokens to predict the next visual token.
  4. Retrieval & Ordering: During inference, the predicted tokens are matched against a candidate pool of images using dot-product similarity to retrieve and arrange the final storyboard.
Overview of the VQ-Trans Model architecture.

Datasets

  • MovieNet-TeViS: A newly introduced benchmark derived from the public MovieNet dataset. It contains 10,000 text synopses, each manually paired by human annotators with an average of 4.6 keyframes chosen for both relevance and cinematic flow. The dataset spans 19 diverse movie genres. Analysis shows it features higher abstractness (lower concreteness scores) and a richer vocabulary compared to standard datasets like LSMDC, MAD, and CMD.

Key Insights and Ideas

  • The Power of Discretization: Quantizing continuous image features into discrete tokens via vector quantization was crucial. It effectively narrows down the search space, allowing the transformer to focus on learning sequential, long-term cinematic transitions rather than getting lost in pixel-level noise.
  • Decoder-Only Prefixes Win: Instead of using a traditional encoder-decoder structure with cross-attention, feeding the text directly as a prefix into a decoder-only architecture yielded significantly better ordering performance.
  • Cinematic Flow Over Simple Relevance: Traditional models like CLIP-Naive look at text-image relevance globally but fail at sequencing. Segmenting text and enforcing sequential constraints (as seen in the CLIP-Sliding and CLIP-Cumulative baselines) drastically improves storyboard coherence.

Key Results

The models were evaluated on two primary tasks using Kendall’s $\tau$ (where 1 is perfect order and -1 is completely reversed):

  • Ordering Shuffled Keyframes: VQ-Trans achieved an overall Kendall’s $\tau$ of 0.367, substantially outperforming the strongest CLIP baseline (CLIP-Cumulative at 0.244) and state-of-the-art story-to-image models like Neural Storyboard (0.163).
  • Retrieve-and-Ordering: When tasked with finding the correct frames out of a 500-image candidate pool and ordering them, VQ-Trans achieved a final metric ($R@K \times \tau@K$ at $K=20$) of 0.300, compared to CLIP-Cumulative's 0.219.
  • The Human Gap: Despite these gains, a massive performance gap remains—human annotators achieved a Kendall’s $\tau$ of 0.821 on the ordering task, underscoring how much room is left for future research.

Limitations and Future Work

While MovieNet-TeViS represents a major step forward, the authors note that the current dataset and model do not explicitly account for complex directorial elements like specific camera angles (e.g., low-angle vs. high-angle), complex camera movements (pans, tilts, dollies), or lighting styles.

Additionally, while the current approach focuses on retrieving existing movie frames, the authors point out that the feature-prediction foundation of VQ-Trans makes it highly extensible to future generative models capable of synthesizing entirely new storyboards from scratch.

Summary updated

Abstract

A video storyboard is a roadmap for video creation which consists of shot-by-shot images to visualize key plots in a text synopsis. Creating video storyboards, however, remains challenging which not only requires cross-modal association between high-level texts and images but also demands long-term reasoning to make transitions smooth across shots. In this paper, we propose a new task called Text synopsis to Video Storyboard (TeViS) which aims to retrieve an ordered sequence of images as the video storyboard to visualize the text synopsis. We construct a MovieNet-TeViS dataset based on the public MovieNet dataset. It contains 10K text synopses each paired with keyframes manually selected from corresponding movies by considering both relevance and cinematic coherence. To benchmark the task, we present strong CLIP-based baselines and a novel VQ-Trans. VQ-Trans first encodes text synopsis and images into a joint embedding space and uses vector quantization (VQ) to improve the visual representation. Then, it auto-regressively generates a sequence of visual features for retrieval and ordering. Experimental results demonstrate that VQ-Trans significantly outperforms prior methods and the CLIP-based baselines. Nevertheless, there is still a large gap compared to human performance suggesting room for promising future work. The code and data are available at: \url{https://ruc-aimind.github.io/projects/TeViS/}
BibTeX
@inproceedings{gu2023tevis,
  author = {Xu Gu and Yuchong Sun and Feiyue Ni and Shizhe Chen and Xihua Wang and Ruihua Song and Boyuan Li and Xiang Cao},
  title = {{TeViS:Translating Text Synopses to Video Storyboards}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612417},
}
Figure