Towards Category Unification of 3D Single Object Tracking on Point Clouds

Jiahao Nie, Zhiwei He, Xudong Lv, Xueyi Zhou, Dong-Kyu Chae, Fei Xie

TL;DR

This paper introduces the first category-unified framework for 3D single object tracking (SOT) on LiDAR point clouds, replacing multiple specialized models with a single network. By adaptively encoding shape and size variations, the proposed SiamCUT and MoCUT models outperform their category-specific counterparts while significantly reducing parameter redundancy.

What Problem Does This Paper Address?

3D Single Object Tracking (SOT) on LiDAR point clouds is a foundational task for autonomous driving, robotics, and augmented reality. Given an object template in the first frame, the tracker must locate it in all subsequent frames.

Currently, state-of-the-art 3D trackers are category-specific. Separate models are trained and deployed independently for cars, pedestrians, vans, and cyclists. This paradigm creates massive parameter redundancy and limits the model's ability to generalize.

Why not just train one model on all categories? The authors discover that naive cross-category training fails due to the extreme variance in object geometry. A massive, boxy truck and a small, slender pedestrian introduce vastly different foreground-to-background ratios in search windows and completely different numerical distributions for bounding-box offset learning. This geometric mismatch destabilizes training and leads to suboptimal performance.

Figure 1: Comparison between different tracking models. In previous category-specific models (a), multiple networks are required to perform individual tracking task for each category. In contrast, our category-unified models (b) can simultaneously track objects across all categories using a single network with shared parameters.

Methods

To bridge the geometric gap between categories, the authors propose a unified framework built on three core designs:

  1. AdaFormer (Unified Representation Network): Standard point cloud backbones use a fixed-radius sphere to group neighboring points, resulting in a static receptive field. AdaFormer introduces a deformable group vector-attention mechanism. A group regression module dynamically predicts scaling and rotation matrices based on local point features, allowing the network's receptive field to adaptively morph to fit different object shapes and sizes.
  2. Unified Model Inputs: Instead of expanding the search area by a fixed physical distance (which introduces excessive background noise for small objects), the search region is scaled relative to the target object's specific dimensions. This keeps the foreground-to-background ratio consistent across all categories.
  3. Unified Learning Objective: To prevent large objects from dominating the regression loss, the authors normalize the 3D spatial offsets by dividing them by the target object's length, width, and height. This maps the regression targets of all categories into a shared numerical space. They also introduce shape-aware labels to maintain a balanced distribution of positive and negative training samples.

The authors integrate these components into both major 3D SOT paradigms, creating SiamCUT (Siamese/appearance-matching paradigm) and MoCUT (motion-centric paradigm).

Figure 2: Overall architecture of AdaFormer. The proposed unified representation network shares a similar three-stage hierarchical structure with existing point set network (Qi et al., 2017b) used in 3D SOT, consisting of a series of subsample operators and AdaFormer blocks. Our representation network is empowered to learn dynamic groups through a deformable group vector-attention sub- block, thereby enabling a variable range of receptive fields.

Datasets

The models were trained and evaluated on two major benchmarks:

  • KITTI: A standard autonomous driving dataset containing 21 training and 29 test sequences.
  • NuScenes: A larger, more challenging dataset containing 700 training and 150 testing scenes.

To evaluate zero-shot generalization capability, the authors also tested their KITTI-trained models directly on the Waymo Open Dataset (WOD) without any fine-tuning.

Key Insights and Ideas

  • Geometry is the Primary Attribute Variance: In 3D point clouds, the lack of color and texture means that category variance boils down to size and shape. By explicitly modeling this geometric variance, category unification becomes possible.
  • Cross-Category Synergy: Training a single model on multiple categories actually improves performance on rarer or data-scarce categories (like pedestrians and cyclists). The model leverages the shared geometric features learned from data-rich categories.
  • Normalization Stabilizes Cross-Category Loss: Mapping bounding box offsets into a dimension-relative space prevents training instability, smoothing out the "jitter" often seen when a single model tries to process multi-category data simultaneously.

Key Results

  • Outperforming Specialized Models: On the KITTI dataset, MoCUT achieved an average Success of 65.8% and Precision of 85.0% using a single model, outperforming its category-specific baseline (M2Track).
  • Massive Gains on Small Objects: SiamCUT boosted the average tracking Success from 42.4% (baseline P2B) to 54.0%—a ~12% absolute gain. Notably, SiamCUT improved Pedestrian tracking Success by 19.5% on KITTI, demonstrating the power of cross-category training synergy.
  • Real-Time Efficiency: Despite the added architectural flexibility, both models maintain real-time inference speeds on a single NVIDIA 3070Ti GPU: SiamCUT runs at 36.4 FPS and MoCUT runs at 47.9 FPS.
  • Strong Generalization: When evaluated on the Waymo Open Dataset, MoCUT consistently outperformed existing state-of-the-art methods across all subsets (Easy, Medium, Hard) for both Vehicle and Pedestrian categories.

Limitations and Future Work

The proposed framework relies on knowing the exact 3D bounding box dimensions of the template target in the initial frame to guide the input scaling and offset normalization. While this is standard for SOT, performance might degrade if the initial template box is noisy or inaccurate. Additionally, the paper primarily focuses on rigid transport categories (cars, pedestrians, cyclists); extending this unification to highly non-rigid or irregular object classes remains an open direction for future research.

Summary updated

Abstract

Category-specific models are provenly valuable methods in 3D single object tracking (SOT) regardless of Siamese or motion-centric paradigms. However, such over-specialized model designs incur redundant parameters, thus limiting the broader applicability of 3D SOT task. This paper first introduces unified models that can simultaneously track objects across all categories using a single network with shared model parameters. Specifically, we propose to explicitly encode distinct attributes associated to different object categories, enabling the model to adapt to cross-category data. We find that the attribute variances of point cloud objects primarily occur from the varying size and shape (e.g., large and square vehicles v.s. small and slender humans). Based on this observation, we design a novel point set representation learning network inheriting transformer architecture, termed AdaFormer, which adaptively encodes the dynamically varying shape and size information from cross-category data in a unified manner. We further incorporate the size and shape prior derived from the known template targets into the model's inputs and learning objective, facilitating the learning of unified representation. Equipped with such designs, we construct two category-unified models SiamCUT and MoCUT.Extensive experiments demonstrate that SiamCUT and MoCUT exhibit strong generalization and training stability. Furthermore, our category-unified models outperform the category-specific counterparts by a significant margin (e.g., on KITTI dataset, 12% and 3% performance gains on the Siamese and motion paradigms). Our code will be available.
BibTeX
@inproceedings{nie2024towards,
  author = {Jiahao Nie and Zhiwei He and Xudong Lv and Xueyi Zhou and Dong-Kyu Chae and Fei Xie},
  title = {{Towards Category Unification of 3D Single Object Tracking on Point Clouds}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=QlqdXrzzD1},
}
Figure