Abstract
Current 3D single object trackers for LiDAR point clouds often fail when encountering object categories not seen during training. This performance degradation stems from spurious correlations between background features and target locations. We present a Feature Decorrelation Method (FDM) that reweights training samples to eliminate statistical dependencies between feature channels. This approach forces the network to learn a general matching function rather than category-specific shortcuts. Our method significantly improves tracking performance on unseen categories in the KITTI and NuScenes datasets, moving closer to robust, class-agnostic autonomous vision.
Introduction
In 3D Single Object Tracking (SOT), a model is tasked with locating a target across subsequent LiDAR point cloud frames given an initial bounding box. While current deep learning trackers like P2B and BAT achieve state-of-the-art results under the Independent and Identically Distributed (I.I.D.) assumption—where training and testing categories match—they struggle in real-world, open-set environments.
When these "class-specific" trackers encounter novel or out-of-distribution (OOD) objects, their performance often plummets. This failure is rooted in the network's tendency to learn spurious correlations between target locations and irrelevant background features specific to the training classes. As shown in Figure 1, even highly advanced trackers like BAT, which excel in class-specific tasks, can perform worse than simpler models when faced with unseen categories.
Methodology: Feature Decorrelation Method (FDM)
To overcome these limitations, we propose a Feature Decorrelation Method (FDM) designed to transform standard Siamese tracking architectures into universal matching functions.
The standard tracking pipeline extracts and fuses features from a "template" (target) and a "search region" using PointNet++. Our FDM intervention treats each channel of these fused features as a distinct variable and seeks to eliminate statistical dependence between them.
Because deep neural features are highly non-linear, measuring dependence is complex. We utilize Random Fourier Features (RFF) to map features into a space where dependence can be quantified via a cross-covariance matrix. We then iteratively learn a set of sample weights that minimize this covariance. By applying these weights to the loss function during training, the network is forced to prioritize true foreground consistency and background discrimination over category-specific shortcuts.
Experimental Setup and Results
We evaluated our approach using the KITTI Tracking Set and NuScenes datasets. To rigorously test generalization, we introduced four new class-agnostic settings. For instance, in "KITTI Setting-1," models are trained on pedestrians, vans, and cyclists, but evaluated solely on cars.
Our results demonstrate that applying FDM (denoted with a "-W" suffix) consistently improves performance over un-modified class-agnostic baselines (denoted with "-U"):
- KITTI: P2B-W improved success and precision by +2.3% and +4.6% respectively over P2B-U. BAT-W saw gains of +3.2% and +3.6% over BAT-U.
- NuScenes: P2B-W achieved a +3.1% success and +4.5% precision improvement over its baseline.
Key Insights
A critical finding of this work is that superior class-specific performance does not guarantee better class-agnostic generalization. Advanced models like BAT, which rely heavily on rigid geometric priors, are particularly prone to overfitting to training shapes.
Qualitative visualizations reveal that standard models often mistakenly cluster background points when tracking unseen classes. Our decorrelation method successfully shifts the network's behavior from "object recognition" to "general matching," effectively highlighting foreground targets while suppressing background clutter.
Conclusion and Future Work
While FDM significantly bridges the gap in class-agnostic tracking, a performance disparity remains compared to fully supervised class-specific models. Future research could explore meta-learning to simulate domain shifts during training or utilize adversarial data generation to further enhance feature robustness. Ultimately, moving beyond category-specific training is essential for the safe deployment of autonomous systems in unpredictable environments.
Summary updated