TL;DR
This paper introduces TeViS, a new task for generating video storyboards from text synopses by retrieving and ordering keyframes. They propose VQ-Trans, a novel decoder-only model that leverages vector quantization for visual representation and auto-regressive generation, achieving state-of-the-art results on their newly created MovieNet-TeViS dataset.
What Problem Does This Paper Address?
Creating professional-looking videos can be challenging for amateurs, as it requires translating a high-level text synopsis into a visual storyboard – a sequence of shot-by-shot images that tell the story coherently. Existing tasks like text-to-video retrieval or generation often focus on short clips with redundant frames or lack the cinematic coherence needed for a storyboard. Story-to-image tasks also don't typically enforce smooth transitions between images. TeViS aims to bridge this gap by retrieving and ordering a sequence of images that accurately and cinematically visualize a given text synopsis, making video creation more accessible.
Methods
The authors propose a two-stage approach:
- Text-to-Image Retrieval: They leverage a pre-trained image-text model (CLIP) to establish an initial connection between the text synopsis and candidate images. This stage involves fine-tuning CLIP to improve retrieval performance.
- Decoder-only Model for Ordering (VQ-Trans): This is the core novel contribution. VQ-Trans uses a decoder-only transformer architecture, inspired by large language models.
- Vector Quantization (VQ): To handle the visual diversity and learn cinematic language, VQ is used to discretize continuous image features into discrete visual tokens. This helps stabilize training and improve generation quality.
- Auto-regressive Generation: The model then auto-regressively generates a sequence of these visual tokens, conditioned on the text synopsis (encoded as a prefix). These tokens are used to retrieve actual images from a candidate pool. The decoder can also predict an "end-of-sequence" token to stop generation.
Datasets
- MovieNet-TeViS: A new benchmark dataset constructed from the public MovieNet dataset. It contains 10,000 text synopses, each paired with an average of 4.6 keyframes manually selected from corresponding movie segments. The selection process emphasizes both relevance to the text and cinematic coherence across frames. This dataset is designed to be more abstract and higher-level than existing datasets like LSMDC and MAD.
Key Insights and Ideas
- The Importance of Cinematic Coherence: The paper highlights that simply retrieving relevant images isn't enough; the sequence must exhibit cinematic coherence, mimicking how a director would storyboard a scene.
- Vector Quantization for Visual Discretization: Using VQ to convert continuous image features into discrete tokens proves crucial for learning the "language of cinematography" and improving the model's ability to generate coherent sequences.
- Decoder-only Architecture for Storyboarding: Adapting a decoder-only transformer architecture, successful in LLMs, allows for flexible handling of variable-length storyboards and effective auto-regressive generation of visual sequences.
- Abstract Synopses Require Different Approaches: The MovieNet-TeViS dataset's synopses are more abstract than typical captions, necessitating models that can perform one-to-many mappings to visualize complex ideas.
Key Results
- Ordering Task: The proposed VQ-Trans model significantly outperforms various CLIP-based baselines and existing story-to-image models, achieving a Kendall's τ of 0.367 on the overall ordering task. This demonstrates its superior ability in sequence modeling and ordering.
- Retrieve-and-Ordering Task: VQ-Trans also achieves the best performance on the more practical retrieve-and-ordering task, outperforming CLIP-based methods and other baselines with an R@K * Kendall's τ@K score of 0.300 for K=20.
- Human Performance Gap: Despite strong results, human performance on the ordering task (Kendall's τ of 0.821) remains significantly higher, indicating substantial room for improvement.
- VQ Effectiveness: Ablation studies show that using Vector Quantization (VQ) significantly improves performance across all metrics.
- Prefix Conditioning Superiority: Using text as a prefix for the decoder-only model is shown to be more effective than cross-attention mechanisms.
Limitations and Future Work
- Limited Movie Style Coverage: The authors acknowledge that the MovieNet-TeViS dataset's coverage of movie styles is not comprehensive, particularly regarding camera angles and movements. Future work will explore these aspects.
- Human Performance Gap: The significant gap between model and human performance suggests that current models still struggle with the nuances of cinematic storytelling.
- Generative Capabilities: The paper mentions that the proposed feature prediction model can be extended to generative models in the future, hinting at the possibility of generating novel frames rather than just retrieving them.
Summary updated