Point Siamese Network for Person Tracking Using 3D Point Clouds

Yubo Cui, Zheng Fang, Sifan Zhou

TL;DR

This paper introduces the Point Siamese Network (PSN), a novel end-to-end deep learning approach for person tracking using only 3D point clouds. PSN overcomes the limitations of traditional filtering methods by directly learning robust features and a similarity map, achieving real-time performance and improved accuracy and robustness, especially in challenging scenarios.

What Problem Does This Paper Address?

Person tracking is crucial for applications like autonomous driving and robot navigation. While visual tracking is common, it struggles with illumination changes and lacks precise distance information. 3D point clouds from LiDAR sensors offer better distance accuracy and robustness to lighting but present unique challenges: they are unordered, making feature extraction difficult for neural networks, and can be sparse, especially at longer distances. Existing 3D tracking methods often rely on Bayesian filtering frameworks, which are sensitive to detection failures and can be less robust in complex scenes with occlusions or fast movements. This paper aims to develop a more robust and accurate 3D person tracking method that doesn't depend on a front-end detector during the tracking phase.

Methods

The core of the proposed method is the Point Siamese Network (PSN), an end-to-end deep neural network designed for 3D point cloud person tracking. PSN operates with two input branches: a "template" branch for the target person's point cloud (initialized from a detector) and a "search" branch for the point cloud in a region where the target might be.

Key components include:

  • Feature Extraction: A PointNet-based network is used to extract local features from each point in the point clouds. This approach is chosen for its ability to handle unordered point sets and its efficiency.
  • Feature Augmentation (Search Branch): To facilitate similarity computation, the features from the search branch are augmented. Each point's feature is repeated to create a "pseudo-cloud," effectively creating an N² x F feature representation.
  • Attention Module (Template Branch): An attention mechanism is applied to the template branch's features. This module learns to weight features, allowing the network to focus on more relevant information and improving robustness, especially in noisy or disturbed scenes.
  • Similarity Map: The augmented search features and weighted template features are used to compute a similarity map. This map highlights points in the search cloud that are most similar to the template. The point with the highest similarity score is identified as the target.
  • End-to-End Tracking: Unlike traditional methods, PSN does not require a detector for every frame. Once initialized, it tracks the person by finding the most similar point in the search cloud. The search box is updated based on the detected target, creating a closed-loop tracking system.

Datasets

  • Training Data: A custom dataset was collected using a Velodyne VLP-16 LiDAR, comprising 8800 frames from 20 different scenes, focusing on open areas. This dataset is publicly released.
  • Evaluation Data:
    • Custom Sequences: Three challenging sequences were collected and labeled, featuring scenarios like "Long Distance," "Fast Rotation," and "Fast Movement." These are also publicly released.
  • KITTI Dataset: The standard KITTI tracking benchmark was used for evaluation.

Key Insights and Ideas

  • Detector-Free Tracking: The most significant insight is demonstrating that robust person tracking can be achieved using only 3D point clouds without relying on a front-end detector for every frame. This directly addresses a major failure point in traditional methods.
  • Attention for Robustness: The attention module in the template branch proves crucial for improving robustness. By adaptively weighting features, it acts as a noise filter, making the tracker more resilient to disturbances and occlusions.
  • Similarity Map for Localization: The use of a similarity map derived from learned features provides a direct and effective way to localize the target within the search region, bypassing the need for complex filtering steps.
  • PointNet for 3D Feature Learning: Leveraging PointNet for feature extraction is a practical choice for handling the unordered nature of point clouds efficiently.

Key Results

  • Accuracy and Robustness: PSN consistently outperformed the Unscented Kalman Filter (UKF) on both custom sequences and the KITTI dataset in terms of accuracy and robustness. For instance, on custom sequences, PSN achieved robustness scores of 97.15%, 96.98%, and 86.13% for the three scenes, significantly higher than UKF's 64.51%, 11.07%, and 26.27%.
  • Real-time Performance: The proposed PSN methods achieved real-time speeds, with frame rates around 14.5 FPS, making them suitable for practical applications.
  • Attention Module Benefits: PSN variants with the attention module (PSN and PSN-mean) showed superior robustness compared to PSN-base (without attention), especially in challenging scenes.
  • Similarity Calculation: The paper explored two methods for calculating the final similarity score (Max+Min vs. Average). The Max+Min approach (Equation 3) was found to be more reasonable and led to better performance.

Limitations and Future Work

  • Initialization Dependency: While PSN is detector-free during tracking, it still requires an initial detection to set up the template. The paper mentions that if the tracker loses the target and the search box becomes empty, the detector is reactivated.
  • Sparsity Impact: The paper notes that in very sparse point cloud scenarios (like some KITTI scenes), the network might have fewer useful features, potentially impacting accuracy compared to denser data.
  • Future Work: The authors explicitly state their plan to extend PSN to multiple people tracking, which is a natural next step given the success of single-person tracking.

Summary updated

Abstract

Person tracking is an important issue in both computer vision and robotics. However, most existing person tracking methods using 3D point cloud are based on the Bayesian Filtering framework which are not robust in challenging scenes. In contrast with the filtering methods, in this paper, we propose a neural network to cope with person tracking using only 3D point cloud, named Point Siamese Network (PSN). PSN consists of two input branches named template and search, respectively. After finding the target person (by reading the label or using a detector), we get the inputs of the two branches and create feature spaces for them using feature extraction network. Meanwhile, a similarity map based on the feature space is proposed between them. We can obtain the target person from the map. Furthermore, we add an attention module to the template branch to guide feature extraction. To evaluate the performance of the proposed method, we compare it with the Unscented Kalman Filter (UKF) on 3 custom labeled challenging scenes and the KITTI dataset. The experimental results show that the proposed method performs better than UKF in robustness and accuracy and has a real-time speed. In addition, we publicly release our collected dataset and the labeled sequences to the research community.
BibTeX
@inproceedings{cui2020point,
  author = {Yubo Cui and Zheng Fang and Sifan Zhou},
  title = {{Point Siamese Network for Person Tracking Using 3D Point Clouds}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=dN0xnexk14r},
}
Figure