Introduction
While text-to-video (T2V) generation has seen significant advancements, providing precise control over object trajectories remains a challenge. Text prompts are often too coarse to describe complex multi-object dynamics, leading to issues with instance consistency and physical plausibility. TrackDiffusion introduces a novel framework that utilizes "tracklets"—sequences of bounding boxes—to provide fine-grained control over object movement, ensuring temporal consistency even in complex scenarios like occlusions and rapid motion.
The Challenge of Precise Animation
Existing T2V models often act as "blunt instruments." For instance, a prompt like "two cars racing" cannot specify the exact arc of a vehicle or the precise moment one passes another. This lack of control manifests in three primary ways:
- Granular Control: Text cannot adequately describe intricate spatial-temporal paths.
- Instance Consistency: Objects frequently "morph" or change appearance as they move across frames.
- Complex Dynamics: Models struggle with re-occurrence (objects leaving and re-entering the frame) and overlapping instances (occlusion).
The TrackDiffusion Framework
TrackDiffusion addresses these limitations by integrating tracklet conditioning into latent diffusion models. The architecture features several key innovations:
- Instance-Aware Location Tokens: Each bounding box is associated with a learnable identity token. This allows the model to distinguish between different instances (e.g., "Dog A" vs. "Dog B") throughout the video sequence.
- Temporal Instance Enhancer (TIE): This module shifts the focus from pixel-level consistency to object-level consistency. By using ROIAlign to crop object features from each frame, the TIE "flattens" motion, allowing the model to maintain a stable representation of an object's appearance regardless of its position.
- Motion Extractor: A self-attention mechanism applied to box coordinates helps the model interpret the intended physics of the trajectory (e.g., acceleration vs. walking).
- Gated Cross-Attention: These layers allow the new control signals to be integrated into pre-trained models without requiring a full retraining from scratch, preserving original generative power while adding trajectory control.
Experimental Results
The researchers evaluated TrackDiffusion using the YouTubeVIS 2021 and MOT-17 datasets. Performance was measured using visual quality (FVD) and a new metric, TrackAP (Tracking Average Precision), which assesses how accurately generated objects follow input trajectories.
TrackDiffusion demonstrated significant improvements over baselines like ModelScope. On YouTubeVIS, it achieved an FVD of 548, compared to ModelScope's 786. Furthermore, it improved TrackAP scores by +3.4 to +9.1 points, indicating much stricter adherence to user-defined paths. Qualitative analysis confirms the model's robustness in handling disappearing and reappearing objects, where standard models typically fail by generating entirely new instances.
Conclusion
TrackDiffusion represents a significant step toward controllable video generation. By prioritizing object-centric features over fixed pixel locations, it achieves superior temporal consistency and trajectory control. Beyond creative applications, the high-quality synthetic data generated by TrackDiffusion has been shown to improve the training of real-world object tracking models, highlighting its potential for the broader computer vision community.
Summary updated