TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models

Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Lanqing Hong, Dit-Yan Yeung, Huchuan Lu, Xu Jia

Introduction

While text-to-video (T2V) generation has seen significant advancements, providing precise control over object trajectories remains a challenge. Text prompts are often too coarse to describe complex multi-object dynamics, leading to issues with instance consistency and physical plausibility. TrackDiffusion introduces a novel framework that utilizes "tracklets"—sequences of bounding boxes—to provide fine-grained control over object movement, ensuring temporal consistency even in complex scenarios like occlusions and rapid motion.

The Challenge of Precise Animation

Existing T2V models often act as "blunt instruments." For instance, a prompt like "two cars racing" cannot specify the exact arc of a vehicle or the precise moment one passes another. This lack of control manifests in three primary ways:

  1. Granular Control: Text cannot adequately describe intricate spatial-temporal paths.
  2. Instance Consistency: Objects frequently "morph" or change appearance as they move across frames.
  3. Complex Dynamics: Models struggle with re-occurrence (objects leaving and re-entering the frame) and overlapping instances (occlusion).

The TrackDiffusion Framework

TrackDiffusion addresses these limitations by integrating tracklet conditioning into latent diffusion models. The architecture features several key innovations:

  • Instance-Aware Location Tokens: Each bounding box is associated with a learnable identity token. This allows the model to distinguish between different instances (e.g., "Dog A" vs. "Dog B") throughout the video sequence.
  • Temporal Instance Enhancer (TIE): This module shifts the focus from pixel-level consistency to object-level consistency. By using ROIAlign to crop object features from each frame, the TIE "flattens" motion, allowing the model to maintain a stable representation of an object's appearance regardless of its position.
  • Motion Extractor: A self-attention mechanism applied to box coordinates helps the model interpret the intended physics of the trajectory (e.g., acceleration vs. walking).
  • Gated Cross-Attention: These layers allow the new control signals to be integrated into pre-trained models without requiring a full retraining from scratch, preserving original generative power while adding trajectory control.
Detailed visualization of TrackDiffusion's tracklet-conditioned generation for a train entering a station.

Experimental Results

The researchers evaluated TrackDiffusion using the YouTubeVIS 2021 and MOT-17 datasets. Performance was measured using visual quality (FVD) and a new metric, TrackAP (Tracking Average Precision), which assesses how accurately generated objects follow input trajectories.

TrackDiffusion demonstrated significant improvements over baselines like ModelScope. On YouTubeVIS, it achieved an FVD of 548, compared to ModelScope's 786. Furthermore, it improved TrackAP scores by +3.4 to +9.1 points, indicating much stricter adherence to user-defined paths. Qualitative analysis confirms the model's robustness in handling disappearing and reappearing objects, where standard models typically fail by generating entirely new instances.

Qualitative comparison between ModelScope and TrackDiffusion across three complex scenarios involving dogs, a cyclist, and trains.

Conclusion

TrackDiffusion represents a significant step toward controllable video generation. By prioritizing object-centric features over fixed pixel locations, it achieves superior temporal consistency and trajectory control. Beyond creative applications, the high-quality synthetic data generated by TrackDiffusion has been shown to improve the training of real-world object tracking models, highlighting its potential for the broader computer vision community.

Summary updated

Abstract

Despite remarkable achievements in video synthesis, achieving granular control over complex dynamics, such as nuanced movement among multiple interacting objects, still presents a significant hurdle for dynamic world modeling, compounded by the necessity to manage appearance and disappearance, drastic scale changes, and ensure consistency for instances across frames. These challenges hinder the development of video generation that can faithfully mimic real-world complexity, limiting utility for applications requiring high-level realism and controllability, including advanced scene simulation and training of perception systems. To address that, we propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control via diffusion models, which facilitates the precise manipulation of the object trajectories and interactions, overcoming the prevalent limitation of scale and continuity disruptions. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects, a critical factor overlooked in the current literature. More-over, we demonstrate that generated video sequences by our TrackDiffusion can be used as training data for visual per-ception models. To the best of our knowledge, this is the first work to apply video diffusion models with tracklet conditions and demonstrate that generated frames can be beneficial for improving the performance of object trackers. 1
BibTeX
@inproceedings{li2025trackdiffusion,
  author = {Pengxiang Li and Kai Chen and Zhili Liu and Ruiyuan Gao and Lanqing Hong and Dit-Yan Yeung and Huchuan Lu and Xu Jia},
  title = {{TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00349},
}
Figure