Efficient Tracking Proposals using 2D-3D Siamese Networks on LIDAR

Jesús Zarzar, Silvio Giancola, Bernard Ghanem

Introduction

Tracking objects in 3D using LIDAR is a fundamental challenge for autonomous systems. Unlike 2D images, LIDAR point clouds are sparse, unstructured, and computationally expensive to search. Traditional state-of-the-art methods typically rely on exhaustive searches—which are too slow for real-time applications—or Bayesian filters like Kalman and Particle filters to estimate object trajectories.

However, these filters often generate poor "proposals" because they rely on simple motion models rather than the actual geometric features of the sensor data. This frequently leads to tracking drift and failure. This paper addresses these limitations by introducing a method to generate a small, highly accurate set of 3D candidates without the massive overhead of searching the entire 3D space.

Proposed Architecture: The Double Siamese Network

The authors propose a Double Siamese Network architecture that decomposes the tracking task into two specialized stages: 2D proposal generation and 3D refinement.

Figure 1: Overview of the Double Siamese Network architecture, showing the transition from 2D BEV proposals to 3D shape refinement and completion.

1. 2D BEV Proposal Generation

The raw 3D point cloud is first projected into a 2D Bird’s Eye View (BEV) image. This representation is dense and structured, enabling the use of standard Convolutional Neural Networks (CNNs). A Region Proposal Network (RPN), built on a 2D Siamese backbone, quickly identifies a small set of likely object locations (candidates) based on visual features rather than just motion priors.

2. 3D Siamese Refinement and Shape Completion

The 2D proposals are converted back into 3D bounding boxes, and the points within are processed by a 3D Siamese network (based on PointNet). This network calculates a similarity score between the candidates and the original object template. To enhance robustness, the network is also trained on a Shape Completion task. Since LIDAR only captures the side of an object facing the sensor, the network learns to reconstruct the full 3D shape, which helps it better distinguish the target from background clutter.

Key Insights

  • Dimensionality Reduction: Searching in 2D (BEV) is significantly more efficient than 3D. While BEV loses some detail, it retains enough information to provide high-quality proposals, allowing the 3D network to focus solely on fine-grained discrimination.
  • Visual Features vs. Motion Priors: Unlike Kalman filters that guess where an object should be, the RPN looks at what the environment actually looks like, allowing the system to recover from abrupt movements.
  • Latent Feature Regularization: By forcing the network to "imagine" the full 3D shape of a vehicle from a partial view, the learned latent features become more descriptive and effective for tracking.

Experimental Results

The method was evaluated on the KITTI dataset, focusing on Cars, Cyclists, and Pedestrians. The results demonstrate a significant performance leap over traditional filtering baselines.

Figure 5: Performance comparison showing Success and Precision curves for the Car class, alongside a table comparing RPN proposals against Kalman (KF) and Particle Filter (PF) baselines.

On the Car class, the method achieved a 12% boost in Success (AUC of IoU) and an 18% boost in Precision (AUC of center distance). Crucially, the RPN-based proposals are so accurate that using just 16 candidates yields better results than using 147 candidates with traditional filtering methods, suggesting a potential 15x speedup in the similarity selection phase.

Limitations and Future Work

While highly effective for rigid objects like cars, the "shape completion" benefit was less pronounced for deformable objects like pedestrians. Because humans change shape significantly while walking, a simple rigid-body assumption is insufficient. Future work may involve incorporating human body pose encoding. Additionally, a "selection gap" remains between the "Best Proposal" and the final tracked result, indicating that the 3D Siamese discriminator can be further refined to better select the optimal candidate from the RPN output.

Summary updated

Abstract

Tracking vehicles in LIDAR point clouds is a challenging task due to the sparsity of the data and the dense search space. The lack of structure in point clouds impedes the use of convolution and correlation filters usually employed in 2D object tracking. In addition, structuring point clouds is cumbersome and implies losing fine-grained information. As a result, generating proposals in 3D space is expensive and inefficient. In this paper, we leverage the dense and structured Bird Eye View (BEV) representation of LIDAR point clouds to efficiently search for objects of interest. We use an efficient Region Proposal Network and generate a small number of object proposals in 3D. Successively, we refine our selection of 3D object candidates by exploiting the similarity capability of a 3D Siamese network. We regularize the latter 3D Siamese network for shape completion to enhance its discrimination capability. Our method attempts to solve both for an efficient search space in the BEV space and a meaningful selection using 3D LIDAR point cloud. We show that the Region Proposal in the BEV outperforms Bayesian methods such as Kalman and Particle Filters in providing proposal by a significant margin and that such candidates are suitable for the 3D Siamese network. By training our method end-to-end, we outperform the previous baseline in vehicle tracking by 12% / 18% in Success and Precision when using only 16 candidates.
BibTeX
@inproceedings{zarzar2019efficient,
  author = {Jesús Zarzar and Silvio Giancola and Bernard Ghanem},
  title = {{Efficient Tracking Proposals using 2D-3D Siamese Networks on LIDAR}},
  booktitle = {King},
  year = {2019},
}
Figure