Abstract
Object tracking remains a fragmented field, with specialized architectures typically required for different tasks such as point tracking, bounding box estimation, or segmentation. We present DINTR (Diffusion-based INterpolation TrackeR), a novel "Tracking-by-Diffusion" paradigm that treats object tracking as a visual interpolation problem. By replacing the computationally expensive mapping to Gaussian noise with a streamlined interpolation process in the latent space, DINTR achieves state-of-the-art performance across five distinct tracking tasks within a single, unified framework.
Introduction
Object tracking is a cornerstone of computer vision, yet current methodologies are often task-specific. Tracking a single point versus a full segmentation mask typically requires disparate architectures and specialized "heads." While generative diffusion models have shown promise, most existing applications use them merely to refine 2D coordinates rather than leveraging the rich visual context inherent in video sequences.
Standard diffusion models generate images from pure Gaussian noise, a process the authors argue is "extensive and unnecessary" for video tracking. Mapping a clear frame to random noise ignores temporal continuity, leading to slower processing and loss of fine-grained context. DINTR addresses this by operating directly in the visual latent space through a diffusion-based interpolation mechanism.
Methodology
The core of DINTR is the Tracking-by-Diffusion approach, which utilizes three key components:
Visual Interpolation
Instead of reconstructing the next frame from scratch, DINTR learns to interpolate between consecutive frames ($I_t$ and $I_{t+1}$). This maintains the model's internal states, ensuring temporal coherence and making the process approximately 2x faster than standard diffusion reconstruction.
Unified Indicators
DINTR employs a conditional mechanism to inject target object information into the generation process. This allows the same core architecture to handle diverse inputs, including points, poses, bounding boxes, segments, or natural language descriptions (e.g., "the red car").
Correspondence Extraction
Localization is achieved by analyzing the model's internal attention maps. By averaging self-attention (capturing object structure) and cross-attention (linking the target prompt to the image), DINTR precisely localizes objects without requiring task-specific decoders.
Experimental Results
DINTR was evaluated across seven benchmarks, including TAP-Vid (points), PoseTrack21 (poses), MOT17/MOTS (boxes/segments), and LaSOT/GroOT (text-based tracking).
Key Performance Metrics
- Point Tracking: Outperformed specialized models like TAPIR on DAVIS and Kinetics datasets.
- Pose Tracking: Achieved 82.5 mAP on PoseTrack21, a significant improvement over previous methods.
- Segmentation: Reached a leading 67.4 sMOTSA on the MOTS benchmark.
- Versatility: DINTR is the first diffusion-based tracker to achieve competitive results across Point, Pose, Box, Segment, and Text representations.
Discussion and Conclusion
The primary insight of DINTR is that interpolation is superior to reconstruction for video tasks. By avoiding "noise-land," the model maintains temporal coherence and remembers object appearances more effectively. Furthermore, the reliance on internal attention maps demonstrates that the generative process naturally highlights object movement.
While DINTR offers significant versatility, limitations remain. There is a performance gap in specific scenarios compared to explicit 2D coordinate motion models (e.g., DiffMOT), and iterative diffusion processes still carry higher computational overhead than simple regression. Future work will explore open-vocabulary tracking and more robust handling of complex camera motions.
Summary updated