Abstract
Multi-object tracking (MOT) is a critical component of autonomous driving systems, necessitating the maintenance of consistent object identities across temporal sequences. While 2D camera images offer high-resolution semantic information, 3D LiDAR point clouds provide accurate spatial geometry. Existing fusion-based MOT methods typically employ "late fusion," which fails to capture low-level spatial-semantic correlations. This paper introduces an Interactive Feature Fusion (IFF) framework that performs multi-scale, bi-directional fusion between 2D and 3D modalities. By allowing pixels and points to mutually query features during extraction, the system achieves a Multiple Object Tracking Accuracy (MOTA) of 84.24% on the KITTI benchmark, demonstrating superior robustness against occlusions and sparse data.
Introduction
Multi-object tracking (MOT) serves as the backbone for situational awareness in autonomous vehicles. The primary challenge lies in maintaining consistent object IDs across frames despite occlusions, varying lighting conditions, and sensor noise. Modern systems leverage both cameras and LiDAR to overcome individual sensor limitations: cameras provide rich texture but lack depth precision, while LiDAR provides accurate 3D structure but is sparse and semantically poor.
Current state-of-the-art methods often treat these sensors as separate streams, fusing their outputs only at the final stages of the pipeline. This "late fusion" approach misses the opportunity to exploit fine-grained spatial relationships that exist at lower levels of feature extraction. This paper proposes a multi-scale, interactive approach to bridge this gap.
Methodology
Overall Architecture
The framework utilizes a VGG-16 backbone for image processing and a modified PointNet++ for point cloud processing. Fusion occurs at multiple scales—from high-resolution, low-level spatial features to low-resolution, high-level semantic features. This multi-scale approach ensures that the model captures both the fine details needed for localization and the abstract features required for robust object identification.
Interactive Feature Fusion (IFF)
The core contribution of this work is the Interactive Feature Fusion (IFF) module. Unlike traditional sequential fusion, IFF enables a bi-directional exchange of information between the 2D and 3D backbones at four distinct stages of the feature extraction process.
The interaction is characterized by two complementary processes:
- Points-centered Interaction: Each 3D point is projected onto the 2D image plane. It then queries and aggregates features from its neighboring pixels using an attention mechanism, enriching the sparse point cloud with dense semantic context.
- Pixels-centered Interaction: Conversely, each pixel queries nearby projected 3D points to incorporate precise spatial and depth information into the image features.
Training Strategy
To optimize the fused system, the authors employed a modality-specific pre-training strategy. The backbones were first trained independently on single-modality data before being integrated and fine-tuned together. This approach provides a strong foundation for the complex interactive learning that follows.
Experimental Results
The proposed model was evaluated on the KITTI MOT dataset, a rigorous benchmark for urban driving scenarios.
Quantitative Performance
The interactive fusion model achieved a MOTA of 84.24% on the KITTI test set, surpassing previous multi-modal methods such as mmMOT (83.23%). More significantly, the method demonstrated a substantial reduction in identity switches (ID-s), recording 415 compared to the baseline's 733. This indicates a much higher level of temporal consistency.
Qualitative Robustness
Visual analysis confirmed that the interactive fusion mechanism allows the tracker to maintain object continuity through challenging scenarios, such as heavy occlusions or large inter-frame displacements, where single-modality or late-fusion models typically fail.
Conclusion
This research demonstrates that early and interactive multi-scale fusion of 2D and 3D data significantly enhances multi-object tracking performance. By establishing a physical and feature-level link between pixels and points, the model achieves greater robustness and accuracy. Future work will focus on optimizing the computational efficiency of the multi-stage fusion process for real-time deployment and exploring learnable spatial alignments between sensors.
Summary updated