Introduction
In professional filmmaking, a storyboard serves as the essential bridge between a written script and the final screen production. It is a roadmap consisting of shot-by-shot images that visualize key plot points. While AI has made significant strides in text-to-video retrieval and story-to-image generation, these existing tasks often fail to capture the "cinematic flow" required for professional storyboarding.
Standard text-to-video models typically focus on short, redundant clips of a single action. Conversely, story-to-image tasks often rely on a rigid one-to-one mapping where every sentence describes exactly one image, leading to sequences that lack narrative transition. To address this gap, researchers have introduced TeViS (Text synopsis to Video Storyboard), a new task that requires models to translate abstract, high-level text synopses into cinematically coherent sequences of retrieved images.
The MovieNet-TeViS Dataset
To benchmark this task, the authors constructed the MovieNet-TeViS dataset, built upon the public MovieNet collection. This specialized dataset contains:
- Scale: 10,000 text synopses paired with 45,584 manually selected keyframes.
- Density: On average, each synopsis is paired with 4.6 keyframes.
- Complexity: Unlike previous datasets such as LSMDC or VIST, MovieNet-TeViS features more abstract text and longer video segments (averaging 63.7 seconds).
This requires models to perform deep reasoning about plot progression and visual transitions—for example, moving from an establishing long shot of a house to a character close-up to reflect a shift in narrative focus.
Methodology: VQ-Trans
The researchers proposed VQ-Trans, a framework designed to reason through the visual flow of a story using three primary components:
- Joint Embedding Space: The model utilizes a fine-tuned CLIP (Contrastive Language-Image Pre-training) model to map both text and images into a shared mathematical space, ensuring that retrieved images are semantically relevant to the synopsis.
- Vector Quantization (VQ): To handle the massive visual diversity found in cinema, VQ-Trans does not process raw image features directly. Instead, a VQ module discretizes continuous visual features into a "codebook" of discrete tokens. This discretization acts as a denoiser, simplifying the visual space and making it easier for the model to learn the "language" of cinematography.
- Auto-regressive Decoder: The model employs a decoder-only Transformer architecture. It treats the text synopsis as a prefix and predicts visual tokens for the storyboard sequentially. This allows the model to consider the "history" of previous shots when choosing the next one, ensuring logical cinematic transitions.
Key Insights and Results
The study highlights that a successful storyboard is defined by cinematic coherence rather than simple relevance. A sequence must flow logically (e.g., establishing shot $\rightarrow$ medium shot $\rightarrow$ close-up). The authors found that treating text as a prefix in a decoder-only model was more effective for this task than traditional cross-attention mechanisms.
Performance was evaluated using Kendall’s $\tau$, a metric measuring sequence order accuracy (where 1.0 is a perfect match):
- Ordering Task: When given the correct images in a shuffled order, VQ-Trans achieved a Kendall’s $\tau$ of 0.367, significantly outperforming CLIP-based baselines (0.244) and specialized story-to-image models like CNSI (0.182).
- Retrieve-and-Order: In the more difficult task of selecting the correct images from a pool of 500 candidates and then ordering them, VQ-Trans maintained its lead with a score of 0.300.
- The Human Gap: Despite these advancements, there remains a significant performance gap. Humans score 0.821 on the ordering task, suggesting that AI still has much to learn regarding the nuances of film direction and visual storytelling.
Conclusion and Future Work
TeViS establishes a new frontier for AI in the creative arts. While VQ-Trans demonstrates the power of vector quantization and auto-regressive reasoning, the authors acknowledge that future models must account for more complex cinematic elements such as specific camera angles, lighting styles, and movement. Furthermore, while this work focuses on retrieval, the VQ-Trans architecture provides a robust foundation for future generative models that could create entirely new storyboard images from scratch.
Summary updated