MLVSNet: Multi-level Voting Siamese Network for 3D Visual Tracking

Zhoutao Wang, Qian Xie, Yu-Kun Lai, Jing Wu, Kun Long, Jun WangICCV 2021 · OpenReview

Abstract

Benefiting from the excellent performance of Siamese-based trackers, huge progress on 2D visual tracking has been achieved. However, 3D visual tracking is still under-explored. Inspired by the idea of Hough voting in 3D object detection, in this paper, we propose a Multi-level Voting Siamese Network (MLVSNet) for 3D visual tracking from outdoor point cloud sequences. To deal with sparsity in outdoor 3D point clouds, we propose to perform Hough voting on multi-level features to get more vote centers and retain more useful information, instead of voting only on the final level feature as in previous methods. We also design an efficient and lightweight Target-Guided Attention (TGA) module to transfer the target information and highlight the target points in the search area. Moreover, we propose a Vote-cluster Feature Enhancement (VFE) module to exploit the relationships between different vote clusters. Extensive experiments on the 3D tracking benchmark of KITTI dataset demonstrate that our MLVSNet outperforms state-of-the-art methods with significant margins.

Introduction

While 2D visual tracking has matured significantly, 3D tracking in outdoor environments remains a major challenge. The primary difficulty lies in the nature of 3D point clouds, which are often irregular, noisy, and extremely sparse in autonomous driving scenarios. Traditional hierarchical backbones like PointNet++ rely on sub-sampling, which can inadvertently discard critical "seed points" necessary for accurate center prediction in sparse scenes. Furthermore, existing feature fusion methods are often computationally expensive or fail to capture global object relationships.

Proposed Architecture

MLVSNet addresses these challenges through three primary architectural innovations: Target-Guided Attention (TGA), Multi-level Voting (MLV), and Vote-cluster Feature Enhancement (VFE).

The overall architecture of MLVSNet, highlighting the TGA, MLV, and VFE modules.

Target-Guided Attention (TGA)

To efficiently merge target and search area features, we introduce the TGA module. Unlike heavy MLP-based fusion, TGA generates a point-wise attention mask. This mask highlights search area points that likely belong to the target, effectively filtering out background noise before the voting stage.

Multi-level Voting (MLV)

The core of MLVSNet is the MLV strategy. Standard Hough voting typically occurs only at the final, most downsampled feature layer. In contrast, MLV performs voting across multiple levels of the feature hierarchy. By aggregating votes from different scales, the network "re-densifies" the decision-making process, ensuring that useful geometric information is preserved even when the input is sparse.

Vote-cluster Feature Enhancement (VFE)

After voting, points are grouped into clusters. The VFE module employs a modified Convolutional Block Attention Module (CBAM) to model spatial and channel relationships between these clusters. This allows the network to perceive the object as a cohesive structure rather than isolated points, leading to more stable and accurate bounding box predictions.

Experimental Evaluation

We evaluated MLVSNet on the KITTI 3D object tracking benchmark, focusing on Car, Pedestrian, Van, and Cyclist categories.

Quantitative Results

MLVSNet achieves state-of-the-art performance, with a mean Success rate of 45.7% and a Precision of 66.6%. This represents a significant improvement over the previous state-of-the-art, P2B, by 3.3% and 6.6%, respectively. Notable gains were observed in the "Van" (+11.2% Success) and "Pedestrian" (+5.4% Success) categories. Despite its multi-level complexity, the model maintains a real-time processing speed of 70 FPS on a single GTX 1080ti.

Qualitative Analysis

Qualitative comparison of MLVSNet against P2B and Ground Truth (GT) on a car tracking sequence. Source, p. 6

Qualitative comparison of MLVSNet against P2B and Ground Truth (GT) on a car tracking sequence.

As shown in the qualitative results, MLVSNet demonstrates superior robustness in maintaining a tight lock on targets throughout a sequence, even in the presence of heavy background noise or when the target consists of very few points.

Conclusion

MLVSNet effectively tackles the sparsity problem in outdoor 3D tracking through multi-level Hough voting and efficient attention mechanisms. While it shows exceptional performance on most categories, future work will focus on improving results for thin objects like cyclists and exploring the integration of RGB data to further distinguish targets from geometrically similar background objects.

Summary updated

Abstract

Benefiting from the excellent performance of Siamese-based trackers, huge progress on 2D visual tracking has been achieved. However, 3D visual tracking is still under-explored. Inspired by the idea of Hough voting in 3D object detection, in this paper, we propose a Multi-level Voting Siamese Network (MLVSNet) for 3D visual tracking from outdoor point cloud sequences. To deal with sparsity in outdoor 3D point clouds, we propose to perform Hough voting on multi-level features to get more vote centers and retain more useful information, instead of voting only on the fi-nal level feature as in previous methods. We also design an efficient and lightweight Target-Guided Attention (TGA) module to transfer the target information and highlight the target points in the search area. Moreover, we propose a Vote-cluster Feature Enhancement (VFE) module to exploit the relationships between different vote clusters. Extensive experiments on the 3D tracking benchmark of KITTI dataset demonstrate that our MLVSNet outperforms state-of-the-art methods with significant margins. Code will be available at https://github.com/CodeWZT/MLVSNet.
BibTeX
@inproceedings{wang2021mlvsnet,
  author = {Zhoutao Wang and Qian Xie and Yu-Kun Lai and Jing Wu and Kun Long and Jun Wang},
  title = {{MLVSNet: Multi-level Voting Siamese Network for 3D Visual Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=TMV-mp5xDz},
}
Figure