ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models

Abstract

ShotAdapter is a lightweight framework that transforms standard, single-shot text-to-video (T2V) models into multi-shot generators capable of producing videos with cinematic cuts. By introducing a learnable "transition token" and a localized attention masking strategy, it allows users to specify the content and duration of individual shots while maintaining consistent character identity across the entire sequence. ShotAdapter requires minimal fine-tuning and generalizes well to longer sequences than those seen during training.

Introduction

While state-of-the-art video diffusion models produce high-quality continuous clips, they struggle with the concept of a "cut." In cinematography, stories are told through multiple shots—different angles, backgrounds, or activities—stitched together. Current T2V models face a "semantic bleeding" problem: when prompted for a sequence of distinct actions, such as a man playing guitar followed by recording on a laptop, the model often blends the two actions into an incoherent blur.

Expand

Source, p. 1

Separately generating clips and concatenating them often results in jarring shifts in character appearance or environmental details. ShotAdapter addresses this by enabling the generation of a single, multi-shot video file that respects distinct shot boundaries without losing character or background consistency.

Methodology

ShotAdapter is built upon Diffusion Transformer (DiT) architectures and utilizes three primary technical innovations:

1. The Transition Token: Inspired by the [EOS] token in natural language processing, a learnable embedding acts as a signal to the model that a shot has ended and a new one is beginning.
2. Local Attention Masking: To prevent inter-shot "pollution," a custom attention mask ensures that textual tokens for a specific shot only attend to the corresponding visual frames. Transition tokens attend specifically to frames where cuts are intended, allowing the model to maintain global identity while localizing specific actions.
3. Lightweight Fine-tuning: The framework requires only approximately 5,000 iterations of fine-tuning—less than 1% of the original pre-training effort—to learn these multi-shot dynamics.

Data Collection and Training

Due to the lack of dedicated multi-shot datasets with consistent identities, the authors developed two pipelines:

• Large Motion Splitting: Long, single-shot videos with significant movement were split into segments and treated as distinct shots.
• Identity Clustering: Videos of the same individual were grouped and concatenated to simulate multi-shot sequences. Precise shot-specific captions were generated using LLaVA-NeXT, and identity stability was verified using YOLO and DINOv2.

Key Results and Insights

ShotAdapter demonstrates remarkable generalizability; models fine-tuned on 2-shot videos can successfully generate 8-shot sequences by repeating the transition token. Quantitative evaluations show that generated cuts occur within 1-2 frames of the user-specified duration.

Expand

Source, p. 1

In comparative studies, ShotAdapter significantly outperformed baselines in Identity Consistency (IC) and Background Consistency (BC). A user study with 75 participants showed an overwhelming preference for ShotAdapter’s results regarding identity consistency and text alignment.

Conclusion and Future Work

ShotAdapter effectively decouples global identity from local action, enabling cinematic multi-shot video generation. Current limitations include a focus on human-centric subjects and constraints on resolution and length imposed by the base T2V architecture. Future work may explore autoregressive approaches to generate full-length short films with consistent characters.

@inproceedings{kara2025shotadapter,
  author = {Ozgur Kara and Krishna Kumar Singh and Feng Liu and Duygu Ceylan and James M. Rehg and Tobias Hinz},
  title = {{ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02645},
}