TL;DR
DINTR introduces a novel "Tracking-by-Diffusion" paradigm that operates directly within the visual latent space, moving away from traditional methods that merely refine 2D bounding box coordinates. By replacing standard Gaussian noise mapping with a specialized latent interpolation mechanism, DINTR achieves a 2x speedup over standard diffusion reconstruction while unifying object tracking across five distinct target representations: points, boxes, segments, poses, and textual prompts.
What Problem Does This Paper Address?
Object tracking is a core computer vision task that requires localizing targets across video frames. While various paradigms exist (such as tracking-by-detection or tracking-by-attention), they often rely on task-specific architectures or separate heads to handle different target modalities (e.g., bounding boxes vs. segmentation masks).
Recently, generative diffusion models have been applied to tracking, but existing approaches primarily operate in the coordinate space—using diffusion mechanics merely to refine the 2D coordinates of bounding boxes. This limits their flexibility and prevents them from expanding to more complex modalities like segments or poses. Conversely, applying standard visual diffusion models directly to video frames is highly inefficient because it requires mapping images entirely to a Gaussian noise domain and back, destroying temporal coherence and incurring massive computational overhead.
DINTR addresses this gap by reformulating diffusion-based tracking to operate entirely in the visual domain using a unified, stable, and fast interpolation process.
Methods
DINTR replaces the traditional diffusion process (which maps data to independent Gaussian noise) with a Temporal Interpolation mechanism. Instead of reconstructing a frame from scratch, the model learns to iteratively blend and transition internal latent states between two consecutive video frames ($I_t$ and $I_{t+1}$).
The core algorithmic contributions include:
- Offset Learning Operator: The authors analyze several interpolative formulations and select a deterministic "learning offset" approach. This operator uses two anchor latents to guide the start and destination points of the transition, providing superior control and stability.
- Correspondence Extraction via Attention: Because the model operates directly on the visual features using a U-Net architecture, the target object's identity is injected as a condition. During the denoising/interpolation steps, the internal self-attention and cross-attention maps inherently capture spatial-temporal correlations. DINTR averages these maps and applies simple mathematical mappings (like
argmaxfor points or thresholding for segments) to extract the final tracking predictions without needing specialized decoder heads. - Multiple-Target Handling: To track multiple objects simultaneously, target representations are concatenated into a single joint input tensor, allowing the model to encode interactions across all targets in a single pass.
Datasets
The authors evaluated DINTR across seven distinct benchmarks to demonstrate its multi-modal versatility:
- TAP-Vid: For long-term physical point tracking.
- PoseTrack21: For human pose estimation and multi-person keypoint tracking.
- DAVIS & MOTS: For single- and multi-target video object segmentation tracking.
- LaSOT & GroOT: For referring expression tracking (tracking objects based on textual prompts).
- MOT17: For standard multi-object bounding box tracking.
Key Insights and Ideas
- Gaussian Noise is Overkill for Tracking: For sequential video data, mapping completely to a Gaussian noise domain is unnecessary and harmful to temporal consistency. Latent interpolation preserves the underlying visual structure, making the optimization landscape much smoother.
- Attention Maps are Natural Localizers: A major "aha" moment is that you don't need complex, task-specific regression heads to output points, boxes, or masks. The internal attention maps of a conditioned visual diffusion model already contain highly precise spatial localization data that can be directly mapped to various outputs.
- Unified Domain Processing: By operating on the core visual domain rather than coordinate spaces, a single model architecture can seamlessly switch between tracking a bounding box, a pixel-level mask, or a human skeleton joint.
Key Results
- Speedup: The proposed interpolation process is 2× faster than standard diffusion-based next-frame reconstruction methods (e.g., reducing interpolation time from 6.2s to 3.2s at 50 steps).
- State-of-the-Art Performance: DINTR achieves top-tier results across multiple modalities:
- Point Tracking: Achieved the highest Average Jaccard (AJ) scores on TAP-Vid datasets (ranging from 57.8 to 85.5).
- Pose Tracking: Scored 82.5 mAP on PoseTrack21, significantly outperforming previous methods like DiffPose (83.0 mAP but on a different heatmap domain) and CorrTrack (72.7 mAP).
- Segment Tracking: Achieved a leading 75.7 J&F score on the DAVIS dataset.
- Text-Based Tracking: Reached a 0.60 precision / 0.58 success rate on LaSOT when tracking via textual prompts without bounding box initializations.
Limitations and Future Work
Despite its strong performance and conceptual elegance, DINTR has a few limitations:
- Motion Modeling Gap: There remains a minor performance gap compared to tracking methods that incorporate explicit, specialized 2D motion models (like DiffMOT) for bounding box tracking on datasets like MOT17.
- Computational Overhead: While twice as fast as standard reconstruction diffusion models, iterative latent interpolation is still computationally heavier than highly optimized heuristic trackers (like ByteTrack), limiting its immediate readiness for real-time edge applications.
Future Directions: The authors suggest exploring open-vocabulary tracking extensions, integrating explicit geometric and camera motion constraints, and optimizing the framework for real-time deployment.
Summary updated