DINTR: Tracking via Diffusion-based Interpolation

Pha A. Nguyen, Ngan Le, Jackson David Cothren, Alper Yilmaz, Khoa Luu

Abstract

Object tracking remains a fragmented field, with specialized architectures typically required for different tasks such as point tracking, bounding box estimation, or segmentation. We present DINTR (Diffusion-based INterpolation TrackeR), a novel "Tracking-by-Diffusion" paradigm that treats object tracking as a visual interpolation problem. By replacing the computationally expensive mapping to Gaussian noise with a streamlined interpolation process in the latent space, DINTR achieves state-of-the-art performance across five distinct tracking tasks within a single, unified framework.

Introduction

Object tracking is a cornerstone of computer vision, yet current methodologies are often task-specific. Tracking a single point versus a full segmentation mask typically requires disparate architectures and specialized "heads." While generative diffusion models have shown promise, most existing applications use them merely to refine 2D coordinates rather than leveraging the rich visual context inherent in video sequences.

Standard diffusion models generate images from pure Gaussian noise, a process the authors argue is "extensive and unnecessary" for video tracking. Mapping a clear frame to random noise ignores temporal continuity, leading to slower processing and loss of fine-grained context. DINTR addresses this by operating directly in the visual latent space through a diffusion-based interpolation mechanism.

DINTR handles diverse tracking tasks (Point, Box, Segment, Text) through a unified diffusion-based interpolation process.

Methodology

The core of DINTR is the Tracking-by-Diffusion approach, which utilizes three key components:

Visual Interpolation

Instead of reconstructing the next frame from scratch, DINTR learns to interpolate between consecutive frames ($I_t$ and $I_{t+1}$). This maintains the model's internal states, ensuring temporal coherence and making the process approximately 2x faster than standard diffusion reconstruction.

Unified Indicators

DINTR employs a conditional mechanism to inject target object information into the generation process. This allows the same core architecture to handle diverse inputs, including points, poses, bounding boxes, segments, or natural language descriptions (e.g., "the red car").

Correspondence Extraction

Localization is achieved by analyzing the model's internal attention maps. By averaging self-attention (capturing object structure) and cross-attention (linking the target prompt to the image), DINTR precisely localizes objects without requiring task-specific decoders.

Experimental Results

DINTR was evaluated across seven benchmarks, including TAP-Vid (points), PoseTrack21 (poses), MOT17/MOTS (boxes/segments), and LaSOT/GroOT (text-based tracking).

Comparison of DINTR against existing tracking paradigms. DINTR is the first to unify all five indication types within a single mechanism.

Key Performance Metrics

  • Point Tracking: Outperformed specialized models like TAPIR on DAVIS and Kinetics datasets.
  • Pose Tracking: Achieved 82.5 mAP on PoseTrack21, a significant improvement over previous methods.
  • Segmentation: Reached a leading 67.4 sMOTSA on the MOTS benchmark.
  • Versatility: DINTR is the first diffusion-based tracker to achieve competitive results across Point, Pose, Box, Segment, and Text representations.

Discussion and Conclusion

The primary insight of DINTR is that interpolation is superior to reconstruction for video tasks. By avoiding "noise-land," the model maintains temporal coherence and remembers object appearances more effectively. Furthermore, the reliance on internal attention maps demonstrates that the generative process naturally highlights object movement.

While DINTR offers significant versatility, limitations remain. There is a performance gap in specific scenarios compared to explicit 2D coordinate motion models (e.g., DiffMOT), and iterative diffusion processes still carry higher computational overhead than simple regression. Future work will explore open-vocabulary tracking and more robust handling of complex camera motions.

Summary updated

Abstract

Object tracking is a fundamental task in computer vision, requiring the localization of objects of interest across video frames. Diffusion models have shown remarkable capabilities in visual generation, making them well-suited for addressing several requirements of the tracking problem. This work proposes a novel diffusion-based methodology to formulate the tracking task. Firstly, their conditional process allows for injecting indications of the target object into the generation process. Secondly, diffusion mechanics can be developed to inherently model temporal correspondences, enabling the reconstruction of actual frames in video. However, existing diffusion models rely on extensive and unnecessary mapping to a Gaussian noise domain, which can be replaced by a more efficient and stable interpolation process. Our proposed interpolation mechanism draws inspiration from classic image-processing techniques, offering a more interpretable, stable, and faster approach tailored specifically for the object tracking task. By leveraging the strengths of diffusion models while circumventing their limitations, our Diffusion-based INterpolation TrackeR (DINTR) presents a promising new paradigm and achieves a superior multiplicity on seven benchmarks across five indicator representations.
BibTeX
@inproceedings{nguyen2024dintr,
  author = {Pha A. Nguyen and Ngan Le and Jackson David Cothren and Alper Yilmaz and Khoa Luu},
  title = {{DINTR: Tracking via Diffusion-based Interpolation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=gAgwqHOBIg},
}
Figure