TL;DR
This paper introduces the first category-unified framework for 3D single object tracking (SOT) on LiDAR point clouds, replacing multiple specialized models with a single network. By adaptively encoding shape and size variations, the proposed SiamCUT and MoCUT models outperform their category-specific counterparts while significantly reducing parameter redundancy.
What Problem Does This Paper Address?
3D Single Object Tracking (SOT) on LiDAR point clouds is a foundational task for autonomous driving, robotics, and augmented reality. Given an object template in the first frame, the tracker must locate it in all subsequent frames.
Currently, state-of-the-art 3D trackers are category-specific. Separate models are trained and deployed independently for cars, pedestrians, vans, and cyclists. This paradigm creates massive parameter redundancy and limits the model's ability to generalize.
Why not just train one model on all categories? The authors discover that naive cross-category training fails due to the extreme variance in object geometry. A massive, boxy truck and a small, slender pedestrian introduce vastly different foreground-to-background ratios in search windows and completely different numerical distributions for bounding-box offset learning. This geometric mismatch destabilizes training and leads to suboptimal performance.
Methods
To bridge the geometric gap between categories, the authors propose a unified framework built on three core designs:
- AdaFormer (Unified Representation Network): Standard point cloud backbones use a fixed-radius sphere to group neighboring points, resulting in a static receptive field. AdaFormer introduces a deformable group vector-attention mechanism. A group regression module dynamically predicts scaling and rotation matrices based on local point features, allowing the network's receptive field to adaptively morph to fit different object shapes and sizes.
- Unified Model Inputs: Instead of expanding the search area by a fixed physical distance (which introduces excessive background noise for small objects), the search region is scaled relative to the target object's specific dimensions. This keeps the foreground-to-background ratio consistent across all categories.
- Unified Learning Objective: To prevent large objects from dominating the regression loss, the authors normalize the 3D spatial offsets by dividing them by the target object's length, width, and height. This maps the regression targets of all categories into a shared numerical space. They also introduce shape-aware labels to maintain a balanced distribution of positive and negative training samples.
The authors integrate these components into both major 3D SOT paradigms, creating SiamCUT (Siamese/appearance-matching paradigm) and MoCUT (motion-centric paradigm).
Datasets
The models were trained and evaluated on two major benchmarks:
- KITTI: A standard autonomous driving dataset containing 21 training and 29 test sequences.
- NuScenes: A larger, more challenging dataset containing 700 training and 150 testing scenes.
To evaluate zero-shot generalization capability, the authors also tested their KITTI-trained models directly on the Waymo Open Dataset (WOD) without any fine-tuning.
Key Insights and Ideas
- Geometry is the Primary Attribute Variance: In 3D point clouds, the lack of color and texture means that category variance boils down to size and shape. By explicitly modeling this geometric variance, category unification becomes possible.
- Cross-Category Synergy: Training a single model on multiple categories actually improves performance on rarer or data-scarce categories (like pedestrians and cyclists). The model leverages the shared geometric features learned from data-rich categories.
- Normalization Stabilizes Cross-Category Loss: Mapping bounding box offsets into a dimension-relative space prevents training instability, smoothing out the "jitter" often seen when a single model tries to process multi-category data simultaneously.
Key Results
- Outperforming Specialized Models: On the KITTI dataset, MoCUT achieved an average Success of 65.8% and Precision of 85.0% using a single model, outperforming its category-specific baseline (M2Track).
- Massive Gains on Small Objects: SiamCUT boosted the average tracking Success from 42.4% (baseline P2B) to 54.0%—a ~12% absolute gain. Notably, SiamCUT improved Pedestrian tracking Success by 19.5% on KITTI, demonstrating the power of cross-category training synergy.
- Real-Time Efficiency: Despite the added architectural flexibility, both models maintain real-time inference speeds on a single NVIDIA 3070Ti GPU: SiamCUT runs at 36.4 FPS and MoCUT runs at 47.9 FPS.
- Strong Generalization: When evaluated on the Waymo Open Dataset, MoCUT consistently outperformed existing state-of-the-art methods across all subsets (Easy, Medium, Hard) for both Vehicle and Pedestrian categories.
Limitations and Future Work
The proposed framework relies on knowing the exact 3D bounding box dimensions of the template target in the initial frame to guide the input scaling and offset normalization. While this is standard for SOT, performance might degrade if the initial template box is noisy or inaccurate. Additionally, the paper primarily focuses on rigid transport categories (cars, pedestrians, cyclists); extending this unification to highly non-rigid or irregular object classes remains an open direction for future research.
Summary updated