SAVTrack: Selective Vote Aggregation for Reliability-Aware Point Cloud Tracking

Sifan Zhou, Linyue Tan, Qiwei Wang, Ziyu Zhao, Xiaobo LuarXiv 2026 · arXiv

TL;DR

SAVTrack introduces a selective vote aggregation mechanism for 3D single object tracking that filters out unreliable point-to-center hypotheses before they can corrupt proposal clustering. By evaluating vote confidence using both local geometry and inter-frame motion context, the framework achieves state-of-the-art point-based tracking performance at a real-time 82 FPS while discarding over 80% of redundant, noisy votes.

What Problem Does This Paper Address?

In 3D Single Object Tracking (SOT), autonomous systems rely on LiDAR point clouds to track targets over time. Point-based trackers often use a "voting" mechanism (inspired by VoteNet), where individual points on an object (seed points) cast votes to predict the object's 3D center.

However, under real-world conditions where point clouds are sparse or incomplete, not all points are equally equipped to guess the center. For instance, points on a flat, featureless surface like a car roof provide weak geometric constraints, leading to highly inaccurate, noisy center votes. Conversely, points on distinct structures like corners or intersections provide rich geometric cues and highly reliable votes.

Figure 1: Illustration of heterogeneous vote reliability across objects based on point clouds. Points on locally planar or ambiguous regions often produce noisy center hypotheses due to weak geometric constraint, while structurally distinctive regions provide more reliable localization and geometric cues. Filtering low-confidence votes before clustering leads to a more concentrated vote distribution and more accurate proposal formation.

Conventional tracking pipelines aggregate all votes indiscriminately. This allows inaccurate hypotheses from ambiguous regions to contaminate the clustering stage, ultimately degrading tracking accuracy.

Methods

The authors propose SAVTrack, a motion-centric tracking framework featuring Selective Vote Aggregation (SAV). Instead of densely aggregating every single vote, SAVTrack filters out low-confidence hypotheses before proposal clustering.

The pipeline operates as follows:

  1. Motion-aware Seed Representation: Features are extracted from consecutive frames using a weight-shared PointNet++ backbone and fused to capture inter-frame motion dynamics.
  2. Sub-region Posterior Estimation: The space around each seed point is partitioned into $N_R$ predefined sub-regions. A lightweight MLP classifier predicts a categorical posterior distribution over these sub-regions, which serves as a proxy for vote reliability.
  3. Region-Specific Vote Regression: Concurrent with classification, region-specific regressors predict the exact offset to the object center.
  4. Hard Pre-Aggregation Gating: Candidate votes with a posterior probability below a threshold ($\tau = 0.3$) are completely discarded. This "hard gating" prevents low-confidence votes from skewing neighborhood construction during the subsequent clustering phase.
  5. Proposal Generation: The remaining high-confidence votes are grouped and clustered to generate the final 3D bounding box prediction.
Figure 2: Overview of SAVTrack. Given the target observation from the previous frame and the current search region, a weight- shared PointNet++ [27] backbone extracts point-wise features, which are fused through the P2P-derived part-to-part motion representation to construct motion-aware seed features. The proposed Selective Vote Aggregation (SAV) module jointly performs center-relative sub-region posterior estimation and region-specific vote regression to generate multiple candidate center hypotheses for each seed. The posterior probability associated with each candidate is used as a proxy for vote reliability, and low-confidence hypotheses are removed by hard pre-aggregation gating before proposal formation. The retained reliable votes are then grouped and aggregated to generate target proposals, from which the final 3D target box is predicted in an end-to-end manner.

Datasets

The model was evaluated on two widely used 3D tracking benchmarks:

  • KITTI: A dataset collected via a 64-beam LiDAR, offering relatively dense point cloud observations across categories like Car, Pedestrian, Van, and Cyclist.
  • nuScenes: A massive dataset captured with a 32-beam LiDAR, presenting significantly sparser target observations and more severe occlusions across Car, Pedestrian, Truck, Trailer, and Bus categories.

Key Insights and Ideas

  • Hard Gating > Soft Weighting: Simply down-weighting bad votes (soft weighting) still allows them to pollute the spatial neighborhood during clustering. Completely removing them (hard gating) ensures cleaner cluster formation and better localization.
  • Motion Matters for Reliability: Estimating vote reliability solely from static, single-frame geometry is insufficient for tracking. Incorporating inter-frame motion context provides crucial temporal clues that significantly improve confidence estimation.
  • Sparsity Resilience: The benefits of selective voting scale inversely with data density. As target observations become sparser, the relative impact of noisy votes increases; filtering them out yields massive performance gains in ultra-sparse scenarios (e.g., targets with fewer than 10 points).

Key Results

SAVTrack achieves an exceptional balance of accuracy and efficiency:

  • KITTI Benchmark: Achieves 68.4% Success and 87.4% Precision, outperforming its baseline (P2P-point) by +2.2% Success and +2.0% Precision.
  • nuScenes Benchmark: Achieves 58.44% Success and 69.82% Precision, showing substantial improvements on challenging large vehicles like Buses (+6.12% Success).
  • Efficiency: Runs at 82 FPS on a single NVIDIA RTX 3090, easily satisfying real-time deployment requirements.
  • Vote Reduction: Retains fewer than one-sixth of the candidate votes used by dense aggregation methods (dropping from ~1536 down to ~250 votes), proving that less is indeed more when it comes to robust proposal generation.

Limitations and Future Work

The authors note a few areas for improvement:

  • Fixed Partitions: SAV currently relies on a fixed, uniformly defined center-relative spatial partition, which might not optimally fit irregularly shaped or highly anisotropic objects. Learning adaptive partitions or coarse-to-fine voting hierarchies could resolve this.
  • Independent Selection: Reliability is currently estimated independently for each seed point. Incorporating set-level consensus or explicit geometric consistency checks among retained hypotheses could further boost robustness against severe occlusions.

Summary updated

Abstract

3D single object tracking (SOT) in LiDAR point clouds is essential for autonomous systems, but remains challenging under sparse and incomplete observations. In such cases, different target points provide highly uneven constraints on the object center, causing some point-to-center votes to be substantially less reliable than others. Existing point-based trackers typically aggregate these hypotheses without explicitly modeling their reliability, allowing inaccurate votes to contaminate proposal clustering and degrade localization accuracy. To address this issue, we propose \textbf{SAVTrack}, a motion-aware tracking framework with \textbf{Selective Vote Aggregation (SAV)}. SAVTrack estimates the reliability of each candidate vote from both local seed features and inter-frame motion context, and removes low-confidence hypotheses before proposal clustering. This pre-aggregation gating prevents unreliable hypotheses from affecting cluster formation while introducing only modest computational overhead. SAVTrack achieves competitive performance on KITTI and nuScenes, reaching 68.4/87.4 and 58.44/69.82 Success/Precision, respectively, while running at 82 FPS. It retains fewer than one-sixth of the candidate votes used by dense aggregation and remains particularly effective under sparse target observations.
BibTeX
@misc{zhou2026savtrack,
  author = {Sifan Zhou and Linyue Tan and Qiwei Wang and Ziyu Zhao and Xiaobo Lu},
  title = {{SAVTrack: Selective Vote Aggregation for Reliability-Aware Point Cloud Tracking}},
  year = {2026},
  eprint = {2609.16662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.16662},
}
Figure