TeViS: Translating Text Synopses to Video Storyboards

Xu Gu, Yuchong Sun, Feiyue Ni, Shizhe Chen, Xihua Wang, Ruihua Song, Boyuan Li, Xiang Cao

TL;DR

This paper introduces TeViS, a new task for generating video storyboards from text synopses by retrieving and ordering keyframes. They propose VQ-Trans, a novel decoder-only model that leverages vector quantization for visual representation and auto-regressive generation, achieving state-of-the-art results on their newly created MovieNet-TeViS dataset.

What Problem Does This Paper Address?

Creating professional-looking videos can be challenging for amateurs, as it requires translating a high-level text synopsis into a visual storyboard – a sequence of shot-by-shot images that tell the story coherently. Existing tasks like text-to-video retrieval or generation often focus on short clips with redundant frames or lack the cinematic coherence needed for a storyboard. Story-to-image tasks also don't typically enforce smooth transitions between images. TeViS aims to bridge this gap by retrieving and ordering a sequence of images that accurately and cinematically visualize a given text synopsis, making video creation more accessible.

Methods

The authors propose a two-stage approach:

  1. Text-to-Image Retrieval: They leverage a pre-trained image-text model (CLIP) to establish an initial connection between the text synopsis and candidate images. This stage involves fine-tuning CLIP to improve retrieval performance.
  2. Decoder-only Model for Ordering (VQ-Trans): This is the core novel contribution. VQ-Trans uses a decoder-only transformer architecture, inspired by large language models.
  3. Vector Quantization (VQ): To handle the visual diversity and learn cinematic language, VQ is used to discretize continuous image features into discrete visual tokens. This helps stabilize training and improve generation quality.
  4. Auto-regressive Generation: The model then auto-regressively generates a sequence of these visual tokens, conditioned on the text synopsis (encoded as a prefix). These tokens are used to retrieve actual images from a candidate pool. The decoder can also predict an "end-of-sequence" token to stop generation.

Datasets

  • MovieNet-TeViS: A new benchmark dataset constructed from the public MovieNet dataset. It contains 10,000 text synopses, each paired with an average of 4.6 keyframes manually selected from corresponding movie segments. The selection process emphasizes both relevance to the text and cinematic coherence across frames. This dataset is designed to be more abstract and higher-level than existing datasets like LSMDC and MAD.

Key Insights and Ideas

  • The Importance of Cinematic Coherence: The paper highlights that simply retrieving relevant images isn't enough; the sequence must exhibit cinematic coherence, mimicking how a director would storyboard a scene.
  • Vector Quantization for Visual Discretization: Using VQ to convert continuous image features into discrete tokens proves crucial for learning the "language of cinematography" and improving the model's ability to generate coherent sequences.
  • Decoder-only Architecture for Storyboarding: Adapting a decoder-only transformer architecture, successful in LLMs, allows for flexible handling of variable-length storyboards and effective auto-regressive generation of visual sequences.
  • Abstract Synopses Require Different Approaches: The MovieNet-TeViS dataset's synopses are more abstract than typical captions, necessitating models that can perform one-to-many mappings to visualize complex ideas.

Key Results

  • Ordering Task: The proposed VQ-Trans model significantly outperforms various CLIP-based baselines and existing story-to-image models, achieving a Kendall's τ of 0.367 on the overall ordering task. This demonstrates its superior ability in sequence modeling and ordering.
  • Retrieve-and-Ordering Task: VQ-Trans also achieves the best performance on the more practical retrieve-and-ordering task, outperforming CLIP-based methods and other baselines with an R@K * Kendall's τ@K score of 0.300 for K=20.
  • Human Performance Gap: Despite strong results, human performance on the ordering task (Kendall's τ of 0.821) remains significantly higher, indicating substantial room for improvement.
  • VQ Effectiveness: Ablation studies show that using Vector Quantization (VQ) significantly improves performance across all metrics.
  • Prefix Conditioning Superiority: Using text as a prefix for the decoder-only model is shown to be more effective than cross-attention mechanisms.

Limitations and Future Work

  • Limited Movie Style Coverage: The authors acknowledge that the MovieNet-TeViS dataset's coverage of movie styles is not comprehensive, particularly regarding camera angles and movements. Future work will explore these aspects.
  • Human Performance Gap: The significant gap between model and human performance suggests that current models still struggle with the nuances of cinematic storytelling.
  • Generative Capabilities: The paper mentions that the proposed feature prediction model can be extended to generative models in the future, hinting at the possibility of generating novel frames rather than just retrieving them.

Summary updated

Abstract

A video storyboard is a roadmap for video creation which consists of shot-by-shot images to visualize key plots in a text synopsis. Creating video storyboards, however, remains challenging which not only requires cross-modal association between high-level texts and images but also demands long-term reasoning to make transitions smooth across shots. In this paper, we propose a new task called Text synopsis to Video Storyboard (TeViS) which aims to retrieve an ordered sequence of images as the video storyboard to visualize the text synopsis. We construct a MovieNet-TeViS dataset based on the public MovieNet dataset [17]. It contains 10K text synopses each paired with keyframes manually selected from corresponding movies by considering both relevance and cinematic coherence. To benchmark the task, we present strong CLIP-based baselines and a novel VQ-Trans model. VQ-Trans first encodes text synopsis and images into a joint embedding space and uses vector quantization (VQ) to improve the visual representation. Then, it auto-regressively generates a sequence of visual features for retrieval and ordering. Experimental results demonstrate that VQ-Trans significantly outperforms prior methods and the CLIP-based baselines. Nevertheless, there is still a large gap compared to human performance suggesting room for promising future work. The code and data are available at: https://ruc-aimind.github.io/projects/TeViS/
BibTeX
@inproceedings{gu2023tevis,
  author = {Xu Gu and Yuchong Sun and Feiyue Ni and Shizhe Chen and Xihua Wang and Ruihua Song and Boyuan Li and Xiang Cao},
  title = {{TeViS: Translating Text Synopses to Video Storyboards}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612417},
}
Figure