Abstract
Benefiting from the excellent performance of Siamese-based trackers, huge progress on 2D visual tracking has been achieved. However, 3D visual tracking is still under-explored. Inspired by the idea of Hough voting in 3D object detection, in this paper, we propose a Multi-level Voting Siamese Network (MLVSNet) for 3D visual tracking from outdoor point cloud sequences. To deal with sparsity in outdoor 3D point clouds, we propose to perform Hough voting on multi-level features to get more vote centers and retain more useful information, instead of voting only on the final level feature as in previous methods. We also design an efficient and lightweight Target-Guided Attention (TGA) module to transfer the target information and highlight the target points in the search area. Moreover, we propose a Vote-cluster Feature Enhancement (VFE) module to exploit the relationships between different vote clusters. Extensive experiments on the 3D tracking benchmark of KITTI dataset demonstrate that our MLVSNet outperforms state-of-the-art methods with significant margins.
Introduction
While 2D visual tracking has matured significantly, 3D tracking in outdoor environments remains a major challenge. The primary difficulty lies in the nature of 3D point clouds, which are often irregular, noisy, and extremely sparse in autonomous driving scenarios. Traditional hierarchical backbones like PointNet++ rely on sub-sampling, which can inadvertently discard critical "seed points" necessary for accurate center prediction in sparse scenes. Furthermore, existing feature fusion methods are often computationally expensive or fail to capture global object relationships.
Proposed Architecture
MLVSNet addresses these challenges through three primary architectural innovations: Target-Guided Attention (TGA), Multi-level Voting (MLV), and Vote-cluster Feature Enhancement (VFE).
Target-Guided Attention (TGA)
To efficiently merge target and search area features, we introduce the TGA module. Unlike heavy MLP-based fusion, TGA generates a point-wise attention mask. This mask highlights search area points that likely belong to the target, effectively filtering out background noise before the voting stage.
Multi-level Voting (MLV)
The core of MLVSNet is the MLV strategy. Standard Hough voting typically occurs only at the final, most downsampled feature layer. In contrast, MLV performs voting across multiple levels of the feature hierarchy. By aggregating votes from different scales, the network "re-densifies" the decision-making process, ensuring that useful geometric information is preserved even when the input is sparse.
Vote-cluster Feature Enhancement (VFE)
After voting, points are grouped into clusters. The VFE module employs a modified Convolutional Block Attention Module (CBAM) to model spatial and channel relationships between these clusters. This allows the network to perceive the object as a cohesive structure rather than isolated points, leading to more stable and accurate bounding box predictions.
Experimental Evaluation
We evaluated MLVSNet on the KITTI 3D object tracking benchmark, focusing on Car, Pedestrian, Van, and Cyclist categories.
Quantitative Results
MLVSNet achieves state-of-the-art performance, with a mean Success rate of 45.7% and a Precision of 66.6%. This represents a significant improvement over the previous state-of-the-art, P2B, by 3.3% and 6.6%, respectively. Notable gains were observed in the "Van" (+11.2% Success) and "Pedestrian" (+5.4% Success) categories. Despite its multi-level complexity, the model maintains a real-time processing speed of 70 FPS on a single GTX 1080ti.
Qualitative Analysis
Qualitative comparison of MLVSNet against P2B and Ground Truth (GT) on a car tracking sequence. Source, p. 6
As shown in the qualitative results, MLVSNet demonstrates superior robustness in maintaining a tight lock on targets throughout a sequence, even in the presence of heavy background noise or when the target consists of very few points.
Conclusion
MLVSNet effectively tackles the sparsity problem in outdoor 3D tracking through multi-level Hough voting and efficient attention mechanisms. While it shows exceptional performance on most categories, future work will focus on improving results for thin objects like cyclists and exploring the integration of RGB data to further distinguish targets from geometrically similar background objects.
Summary updated