TL;DR
The paper introduces TeViS (Text synopsis to Video Storyboard), a novel multimodal task aimed at translating abstract text synopses into cinematically coherent sequences of keyframes. To benchmark this task, the authors release the MovieNet-TeViS dataset and propose VQ-Trans, an auto-regressive decoder-only transformer that leverages vector quantization to significantly outperform traditional cross-modal retrieval methods.
What Problem Does This Paper Address?
When directors plan a film, they don't just look for images that match a script; they design a storyboard—a sequence of shots that flow together cinematically (e.g., transitioning from an establishing long shot to a dramatic close-up).
Existing text-to-vision methods fall short here. Text-to-video retrieval focuses on short, isolated clips with highly redundant frames. Meanwhile, story-to-image retrieval relies on a strict one-to-one mapping from detailed sentences to individual images, completely ignoring the fluid "lens language" required for video creation.
The TeViS task challenges models to take a brief, high-level text synopsis and retrieve an ordered, cinematically smooth sequence of images from a massive movie database, effectively acting as an automated assistant for amateur video creators.
Methods
The authors propose VQ-Trans, a framework built on a decoder-only transformer architecture designed to handle both image ordering and variable-length sequence generation.
- Text Prefix Encoding: The input text synopsis is encoded using a fine-tuned CLIP text encoder. These text features serve as a "prefix" hint for the transformer decoder.
- Image Discretization via Vector Quantization (VQ): Learning the language of cinematography from continuous, highly diverse image features is notoriously difficult. To stabilize training, VQ-Trans uses a fine-tuned CLIP image encoder combined with a VQ codebook ($C$) to quantize continuous visual features into discrete visual tokens.
- Auto-regressive Generation: The transformer decoder ingests the text prefix and the history of previous discrete image tokens to predict the next visual token.
- Retrieval & Ordering: During inference, the predicted tokens are matched against a candidate pool of images using dot-product similarity to retrieve and arrange the final storyboard.
Datasets
- MovieNet-TeViS: A newly introduced benchmark derived from the public MovieNet dataset. It contains 10,000 text synopses, each manually paired by human annotators with an average of 4.6 keyframes chosen for both relevance and cinematic flow. The dataset spans 19 diverse movie genres. Analysis shows it features higher abstractness (lower concreteness scores) and a richer vocabulary compared to standard datasets like LSMDC, MAD, and CMD.
Key Insights and Ideas
- The Power of Discretization: Quantizing continuous image features into discrete tokens via vector quantization was crucial. It effectively narrows down the search space, allowing the transformer to focus on learning sequential, long-term cinematic transitions rather than getting lost in pixel-level noise.
- Decoder-Only Prefixes Win: Instead of using a traditional encoder-decoder structure with cross-attention, feeding the text directly as a prefix into a decoder-only architecture yielded significantly better ordering performance.
- Cinematic Flow Over Simple Relevance: Traditional models like CLIP-Naive look at text-image relevance globally but fail at sequencing. Segmenting text and enforcing sequential constraints (as seen in the CLIP-Sliding and CLIP-Cumulative baselines) drastically improves storyboard coherence.
Key Results
The models were evaluated on two primary tasks using Kendall’s $\tau$ (where 1 is perfect order and -1 is completely reversed):
- Ordering Shuffled Keyframes: VQ-Trans achieved an overall Kendall’s $\tau$ of 0.367, substantially outperforming the strongest CLIP baseline (CLIP-Cumulative at 0.244) and state-of-the-art story-to-image models like Neural Storyboard (0.163).
- Retrieve-and-Ordering: When tasked with finding the correct frames out of a 500-image candidate pool and ordering them, VQ-Trans achieved a final metric ($R@K \times \tau@K$ at $K=20$) of 0.300, compared to CLIP-Cumulative's 0.219.
- The Human Gap: Despite these gains, a massive performance gap remains—human annotators achieved a Kendall’s $\tau$ of 0.821 on the ordering task, underscoring how much room is left for future research.
Limitations and Future Work
While MovieNet-TeViS represents a major step forward, the authors note that the current dataset and model do not explicitly account for complex directorial elements like specific camera angles (e.g., low-angle vs. high-angle), complex camera movements (pans, tilts, dollies), or lighting styles.
Additionally, while the current approach focuses on retrieving existing movie frames, the authors point out that the feature-prediction foundation of VQ-Trans makes it highly extensible to future generative models capable of synthesizing entirely new storyboards from scratch.
Summary updated