DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models

Xunjie He, Christina Dao Wen Lee, Meiling Wang, Chengran Yuan, Zefan Huang, Yufeng Yue, Marcelo H. Ang Jr.arXiv 2025 · DOI · arXiv

Abstract

Collaborative perception enables multiple agents to share sensor data to overcome individual limitations like occlusions. However, existing methods primarily focus on single-class (vehicle) detection. This paper introduces DINO-CoDT, the first collaborative framework for multi-class 3D detection and tracking of vehicles, pedestrians, and trucks. By leveraging a Vision Foundation Model (DINOv2) for semantic re-identification and an adaptive attention mechanism, DINO-CoDT addresses challenges like scale variance and identity switches for small road users. Evaluations on the V2X-Real dataset demonstrate significant improvements in both detection accuracy and tracking stability.

Introduction

Collaborative perception allows vehicles and infrastructure to exchange sensor information, effectively expanding their field of view and eliminating blind spots. Despite its potential, current research is heavily biased toward vehicle detection. Real-world urban environments are heterogeneous, populated by pedestrians, cyclists, and trucks with diverse scales and motion patterns.

Detecting and tracking these diverse classes presents three major challenges:

  1. Scale Variance: Small objects like pedestrians have sparse LiDAR representations in Bird’s Eye View (BEV), leading to frequent misses.
  2. Identity Switches: Geometric association (e.g., IoU) often fails for small or erratically moving objects.
  3. Asynchronous Observations: Fusing data from multiple agents with different viewpoints and timestamps is complex.

Methods

DINO-CoDT addresses these issues through three core components integrated into a unified pipeline (see Figure 1).

DINO-CoDT Architecture Overview

Global Spatial Attention Fusion (GSAF)

This module employs multi-scale attention to capture fine-grained details for small objects and broad context for larger ones. It dynamically weights features from different agents, prioritizing the most relevant data for fusion.

Tracklet RE-IDentification (REID) with DINOv2

To mitigate identity switches when geometric tracking fails, the system utilizes DINOv2, a powerful Vision Foundation Model. High-dimensional semantic "fingerprints" are extracted from 2D image crops to match objects across cameras and time. Interestingly, DINOv2 can be used "off-the-shelf" without domain-specific fine-tuning; its 31st transformer layer provides sufficient semantic richness to distinguish between similar-looking pedestrians across different viewpoints.

Velocity-based Adaptive Tracklet Management (VATM)

Unlike standard trackers with fixed thresholds, VATM adjusts "death" thresholds based on object velocity. This allows the system to be more persistent with slow-moving, erratic pedestrians compared to fast-moving vehicles.

Results

The model was evaluated on the V2X-Real dataset, a large-scale real-world benchmark for multi-class collaborative perception, and the OPV2V simulation dataset.

DINO-CoDT establishes a new state-of-the-art on V2X-Real. As shown in Figure 2, it achieves a significant performance boost over previous methods like V2X-ViT. Specifically, it reaches 68.7% [email protected] for vehicles and 41.5% for pedestrians, representing a 6-9% improvement across classes.

Detection Performance on V2X-Real Dataset

In terms of tracking, the model significantly improves the sAMOTA metric for pedestrians from 18.37% to 23.75%. Ablation studies confirm that the DINOv2-based REID module is critical for reducing identity switches (IDSW), proving that visual semantic memory is far more reliable than geometry for small objects in noisy BEV spaces.

Conclusion and Future Work

DINO-CoDT demonstrates that integrating Vision Foundation Models into collaborative perception frameworks can drastically improve the detection and tracking of diverse road users. While challenges remain regarding communication bandwidth and the need for decentralized architectures, this work provides a robust foundation for safer multi-agent autonomous systems. Future research will explore open-vocabulary perception to detect objects beyond the predefined training classes.

Summary updated

Abstract

Collaborative perception plays a crucial role in enhancing environmental understanding by expanding the perceptual range and improving robustness against sensor failures, which primarily involves collaborative 3D detection and tracking tasks. The former focuses on object recognition in individual frames, while the latter captures continuous instance tracklets over time. However, existing works in both areas predominantly focus on the vehicle superclass, lacking effective solutions for both multi-class collaborative detection and tracking. This limitation hinders their applicability in real-world scenarios, which involve diverse object classes with varying appearances and motion patterns. To overcome these limitations, we propose a multi-class collaborative detection and tracking framework tailored for diverse road users. We first present a detector with a global spatial attention fusion (GSAF) module, enhancing multi-scale feature learning for objects of varying sizes. Next, we introduce a tracklet RE-IDentification (REID) module that leverages visual semantics with a vision foundation model to effectively reduce ID SWitch (IDSW) errors, in cases of erroneous mismatches involving small objects like pedestrians. We further design a velocity-based adaptive tracklet management (VATM) module that adjusts the tracking interval dynamically based on object motion. Extensive experiments on the V2X-Real and OPV2V datasets show that our approach significantly outperforms existing state-of-the-art methods in both detection and tracking accuracy.
BibTeX
@misc{he2025dino,
  author = {Xunjie He and Christina Dao Wen Lee and Meiling Wang and Chengran Yuan and Zefan Huang and Yufeng Yue and Marcelo H. Ang Jr.},
  title = {{DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models}},
  year = {2025},
  eprint = {2506.07375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07375},
}
Figure