Introduction
Deep learning has revolutionized autonomous driving, particularly in the realm of 3D object detection using LiDAR. However, the field remains constrained by two significant bottlenecks: the immense labor cost of data annotation and the persistent challenge of "corner cases." Annotating a single 3D LiDAR frame can take an expert approximately 10 minutes; scaling this to a standard one-hour sequence would require over 1,000 days of manual labor. Furthermore, real-world datasets are naturally skewed toward common objects like cars and pedestrians, leaving models vulnerable to rare but critical hazards, such as uniquely oriented motorcycles or specialized construction vehicles.
While simulators like CARLA offer a solution by generating infinite labeled data, the "sim-to-real gap" prevents synthetic data from being a direct substitute. Differences in sensor noise and data structures (e.g., point intensity) often cause models trained on simulation to perform poorly in the real world. This paper introduces JiSAM, a novel framework designed to bridge this gap, allowing 3D object detectors to achieve state-of-the-art performance using only 2.5% of available real-world labels.
Methodology
JiSAM addresses the simulation-to-real gap through three "plug-and-play" components that align synthetic features with real-world distributions.
Jittering Augmentation (JA)
Traditional 3D augmentations typically operate in Cartesian (XYZ) space. JiSAM introduces Jittering Augmentation, which adds random noise to synthetic points in Spherical Coordinates ($r, \theta, \phi$). This approach more accurately mimics the physical uncertainty inherent in real LiDAR laser pulses, such as range errors and angular resolution limits. By making synthetic data more representative of physical sensor noise, JA significantly improves the sample efficiency of simulation data.
Domain-Aware Backbone
Real-world LiDAR sensors often provide metadata, such as intensity and timestamps, which are difficult for simulators to replicate accurately. To prevent these discrepancies from confusing the model, JiSAM employs a Domain-Aware Backbone. It utilizes separate, lightweight input embedding layers for the real and synthetic domains. These layers process domain-specific features before passing the data into a shared, high-capacity backbone, ensuring the model learns robust features without being misled by simulation artifacts.
Sectorized AlignMent (SMA)
SMA is the core innovation of the JiSAM framework. The authors observed that the point distribution of a LiDAR-detected object is highly dependent on its geometric relationship to the sensor—specifically its distance (sector) and orientation (heading). SMA maintains a Memory Bank that stores features of real objects categorized by these geometric sectors. During training, the framework forces the features of synthetic objects to align with the corresponding "real" features in the memory bank. This granular, geometry-based alignment is far more effective than previous global alignment strategies.
Experimental Results
The researchers evaluated JiSAM using the NuScenes dataset for real-world validation and CARLA for synthetic data generation (80,000 frames).
Label Efficiency
The results demonstrate remarkable efficiency. By utilizing only 2.5% of real labels (approximately 7,000 frames) combined with synthetic data, JiSAM achieved a performance of 63.95 mAP. This is nearly identical to the performance of the state-of-the-art Transfusion model (64.51 mAP) when trained on 100% of the real labels. Furthermore, JiSAM outperformed standard semi-supervised learning methods by approximately 4 mAP and 3 NDS (NuScenes Detection Score) under the same limited-data conditions.
Solving the Corner Case Problem
One of the most significant findings involves zero-shot detection of rare objects. In an experiment where all motorcycle labels were removed from the real-world training set, JiSAM achieved 16% mAP on real motorcycles. By learning from synthetic motorcycles in CARLA and aligning those features with the general real-world feature space via SMA, the model successfully detected objects it had never seen in a real-world labeled context. In contrast, standard models failed completely in this scenario.
Conclusion and Future Work
JiSAM provides a powerful framework for reducing the data dependency of 3D object detectors while simultaneously addressing the safety-critical issue of corner cases. By focusing on geometric alignment and realistic noise modeling, it bridges the sim-to-real gap more effectively than previous methods.
While the framework shows immense promise, the authors note that performance on certain categories like bicycles remains slightly lower than full-label models, likely due to a lack of asset diversity in current simulators. Future work may involve combining JiSAM with emerging LiDAR generative models to further diversify synthetic training sets and narrow the remaining performance gaps.
Summary updated