Abstract
Collaborative perception enables multiple agents to share sensor data to overcome individual limitations like occlusions. However, existing methods primarily focus on single-class (vehicle) detection. This paper introduces DINO-CoDT, the first collaborative framework for multi-class 3D detection and tracking of vehicles, pedestrians, and trucks. By leveraging a Vision Foundation Model (DINOv2) for semantic re-identification and an adaptive attention mechanism, DINO-CoDT addresses challenges like scale variance and identity switches for small road users. Evaluations on the V2X-Real dataset demonstrate significant improvements in both detection accuracy and tracking stability.
Introduction
Collaborative perception allows vehicles and infrastructure to exchange sensor information, effectively expanding their field of view and eliminating blind spots. Despite its potential, current research is heavily biased toward vehicle detection. Real-world urban environments are heterogeneous, populated by pedestrians, cyclists, and trucks with diverse scales and motion patterns.
Detecting and tracking these diverse classes presents three major challenges:
- Scale Variance: Small objects like pedestrians have sparse LiDAR representations in Bird’s Eye View (BEV), leading to frequent misses.
- Identity Switches: Geometric association (e.g., IoU) often fails for small or erratically moving objects.
- Asynchronous Observations: Fusing data from multiple agents with different viewpoints and timestamps is complex.
Methods
DINO-CoDT addresses these issues through three core components integrated into a unified pipeline (see Figure 1).
Global Spatial Attention Fusion (GSAF)
This module employs multi-scale attention to capture fine-grained details for small objects and broad context for larger ones. It dynamically weights features from different agents, prioritizing the most relevant data for fusion.
Tracklet RE-IDentification (REID) with DINOv2
To mitigate identity switches when geometric tracking fails, the system utilizes DINOv2, a powerful Vision Foundation Model. High-dimensional semantic "fingerprints" are extracted from 2D image crops to match objects across cameras and time. Interestingly, DINOv2 can be used "off-the-shelf" without domain-specific fine-tuning; its 31st transformer layer provides sufficient semantic richness to distinguish between similar-looking pedestrians across different viewpoints.
Velocity-based Adaptive Tracklet Management (VATM)
Unlike standard trackers with fixed thresholds, VATM adjusts "death" thresholds based on object velocity. This allows the system to be more persistent with slow-moving, erratic pedestrians compared to fast-moving vehicles.
Results
The model was evaluated on the V2X-Real dataset, a large-scale real-world benchmark for multi-class collaborative perception, and the OPV2V simulation dataset.
DINO-CoDT establishes a new state-of-the-art on V2X-Real. As shown in Figure 2, it achieves a significant performance boost over previous methods like V2X-ViT. Specifically, it reaches 68.7% [email protected] for vehicles and 41.5% for pedestrians, representing a 6-9% improvement across classes.
In terms of tracking, the model significantly improves the sAMOTA metric for pedestrians from 18.37% to 23.75%. Ablation studies confirm that the DINOv2-based REID module is critical for reducing identity switches (IDSW), proving that visual semantic memory is far more reliable than geometry for small objects in noisy BEV spaces.
Conclusion and Future Work
DINO-CoDT demonstrates that integrating Vision Foundation Models into collaborative perception frameworks can drastically improve the detection and tracking of diverse road users. While challenges remain regarding communication bandwidth and the need for decentralized architectures, this work provides a robust foundation for safer multi-agent autonomous systems. Future research will explore open-vocabulary perception to detect objects beyond the predefined training classes.
Summary updated