Abstract
While modern text-to-video (T2V) models excel at generating high-quality single-shot clips, they frequently fail to execute narrative transitions requested in complex prompts. This paper identifies the "one-scene trap" caused by a lack of transition-rich training data. To bridge this gap, the authors introduce the Transition-Aware Video (TAV) dataset and a targeted post-training pipeline. Results demonstrate that fine-tuning with structured temporal prompts enables models to recognize scene boundaries, increasing the average number of generated scenes from approximately 1.1 to nearly 3.0 without compromising aesthetic quality.
The "One-Scene Trap" in Video Generation
Current state-of-the-art open-source video models often ignore temporal connectors like "then" or "next." For instance, a prompt requesting "a girl standing by a tree, then a painting of a forest" typically results in a video of only the first scene. This limitation stems from a data bottleneck: over 90% of clips in standard datasets like WebVid-10M and Panda-70M are single-scene. Consequently, models do not learn the visual "grammar" of a cut or transition.
Methodology: Teaching Transitions
The researchers developed a pipeline to curate the TAV dataset and fine-tune models for transition awareness:
- Scene Detection: Using a modified
PySceneDetect, the team identified hard cuts in existing videos by analyzing pixel differences across HSV channels. - Clip Extraction: 10-second segments were extracted, centered on detected transitions (5 seconds per scene).
- Structured Captioning: Descriptions for each scene were generated via the BLIP model and fused into a specific template:
"{Previous scene: [Description A]; Next scene: [Description B]}". - Post-Training: The OpenSora-Plan v1.3.1 model was fine-tuned on the 500-pair TAV dataset using a single H200 GPU.
Quantitative and Qualitative Results
The impact of post-training was evaluated using VBench and custom multi-scene prompts. As shown in the performance metrics, the baseline models consistently failed to transition, typically averaging only slightly more than one scene per video.
After 24 epochs of post-training, the model's ability to generate multiple scenes improved dramatically, reaching an average of 2.9 scenes for structured prompts (Group C). Qualitatively, the model shifted from attempting to blend disparate descriptions into a single frame to executing clean, logical cuts.
Key Insights
- Conceptual Generalization: Training on explicit "Previous/Next" templates (Group C) improved performance even on natural language prompts (Group B), suggesting the model learned the underlying concept of a transition rather than just keyword matching.
- Quality Preservation: Post-training did not lead to "catastrophic forgetting." Aesthetic and imaging quality remained stable, while "dynamic degrees"—a measure of motion coherence—actually improved.
Limitations and Future Outlook
This preliminary study utilized a relatively small dataset of 500 videos. Future work aims to scale the TAV dataset to thousands of transitions and explore more complex, multi-cut narrative structures. Additionally, replacing heuristic-based scene detection with AI-driven methods could further enhance training data quality.
Summary updated