Point-Syn2Real: Semi-Supervised Synthetic-to-Real Cross-Domain Learning for Object Classification in 3D Point Clouds

Ziwei Wang, Reza Arablouei, Jiajun Liu, Paulo Borges, Greg Bishop-Hurley, Nicholas Heaney

TL;DR

Point-Syn2Real is a new framework that trains 3D object classifiers using "free" synthetic CAD models and unlabeled real-world data, bypassing the need for expensive manual point cloud annotation. By simulating realistic LiDAR occlusions and using semi-supervised entropy minimization, the approach achieves state-of-the-art performance in transferring knowledge from simulation to reality.

What Problem Does This Paper Address?

Deep learning models for 3D point clouds (like those used in autonomous driving) are data-hungry. However, labeling 3D data is a "labor-intensive nightmare"—human annotators must rotate and inspect sparse clusters of dots from multiple angles to identify objects.

While we have massive libraries of 3D CAD models (synthetic data), models trained on perfect 3D shapes often fail when deployed on real LiDAR sensors. This is due to the domain gap: real LiDAR scans are noisy, sparse, and only capture the side of the object facing the sensor (partial occlusion), whereas CAD models are complete and "clean."

Methods

The authors propose a three-pillar approach to bridge the synthetic-to-real gap:

  1. Realistic Multi-view Simulation: Instead of using complete 3D models, they use Blender to simulate how a LiDAR sensor actually "sees." They place a virtual sensor at random positions around a CAD model, generate a depth map, and back-project it into a point cloud. This creates realistic, partially-occluded views.
  2. Robust Data Augmentation: To handle the variability of the real world, they apply random z-axis rotations and add Gaussian noise to the synthetic samples. This teaches the model to be invariant to an object's orientation and sensor imperfections.
  3. Semi-Supervised Learning (Entropy Minimization): This is the "secret sauce" for the real-world domain. The model is trained on labeled synthetic data using a weighted cross-entropy loss. Simultaneously, it processes unlabeled real-world data. The training objective encourages the model to make "confident" predictions on this unlabeled data by minimizing its Shannon entropy. Essentially, it forces the model to decide on a class rather than remaining uncertain.

The architecture is flexible; the authors successfully tested it using both DGCNN and PointNet++ as backbone encoders.

Datasets

The authors introduced a new benchmark dataset called Point-Syn2Real, which covers two distinct scenarios:

  • Indoor: Uses ModelNet and ShapeNet as the synthetic source and ScanNet (real-world room scans) as the target.
  • Outdoor: Uses a newly curated 3D_City subset (from 3D Warehouse and ShapeNet) as the source and SemanticKITTI (real-world driving data) as the target.

Key Insights and Ideas

  • Occlusion is a Feature, Not a Bug: The most critical insight is that simulating the physics of the sensor (partial views) is more important than having high-fidelity textures. By training on "broken" versions of CAD models, the model learns features that actually appear in real LiDAR scans.
  • Entropy as a Bridge: Entropy minimization acts as a form of self-training. It allows the model to look at real-world data it has never seen labels for and say, "I'm not sure what this is, but it looks most like a 'Car,' so I will refine my features to be more certain it's a 'Car'."
  • Simplicity Wins: Unlike complex Domain Adversarial Neural Networks (DANN) that try to "confuse" a discriminator, this approach relies on smart data preparation and a standard semi-supervised loss, making it easier to implement and stabilize.

Key Results

The proposed method significantly outperformed existing domain adaptation techniques:

  • Outdoor Performance: In the 3D_City to SemanticKITTI task, the baseline (no adaptation) achieved a dismal 19.76% accuracy. The full Point-Syn2Real (A+S+E) reached 96.57% accuracy, nearly matching the performance of a fully supervised model (97.06%).
  • Indoor Performance: On the ModelNet to ScanNet task, the method achieved 59.13% accuracy, outperforming specialized domain adaptation methods like PointDAN (33.00%) and DefRec+PCM (51.80%).
  • Backbone Agnostic: The results held steady whether using DGCNN or PointNet++, proving the framework's versatility.

Limitations and Future Work

  • Class Imbalance: The authors noted that the outdoor dataset is highly imbalanced (reflecting real-world driving where "Cars" are more common than "Motorcycles"). While they used weighted loss functions, very rare classes still pose a challenge.
  • CAD Dependency: The method relies on the availability of 3D CAD models for the target classes. If an object doesn't exist in a synthetic library, the model cannot learn to recognize it.
  • Complex Scenes: The current work focuses on object classification (identifying a cropped object). Future work could extend this to semantic segmentation (labeling every point in a massive, complex scene).

Summary updated

Abstract

Object classification using LiDAR 3D point cloud data is critical for modern applications such as autonomous driving. However, labeling point cloud data is labor-intensive as it requires human annotators to visualize and inspect the 3D data from different perspectives. In this paper, we propose a semi-supervised cross-domain learning approach that does not rely on manual annotations of point clouds and performs similar to fully-supervised approaches. We utilize available 3D object models to train classifiers that can generalize to real-world point clouds. We simulate the acquisition of point clouds by sampling 3D object models from multiple viewpoints and with arbitrary partial occlusions. We then augment the resulting set of point clouds through random rotations and adding Gaussian noise to better emulate the real-world scenarios. We then train point cloud encoding models on the synthesized and augmented datasets and evaluate their cross-domain classification performance on corresponding real-world datasets. We also introduce PointSyn2Real, a new benchmark dataset for cross-domain learning on point clouds. The results of our extensive experiments with this dataset demonstrate that the proposed cross-domain learning approach for point clouds outperforms the related baseline and state-of-the-art approaches in both indoor and outdoor settings in terms of cross-domain generalizability. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@inproceedings{wang2023point,
  author = {Ziwei Wang and Reza Arablouei and Jiajun Liu and Paulo Borges and Greg Bishop-Hurley and Nicholas Heaney},
  title = {{Point-Syn2Real: Semi-Supervised Synthetic-to-Real Cross-Domain Learning for Object Classification in 3D Point Clouds}},
  booktitle = {ICME},
  year = {2023},
  url = {https://openreview.net/forum?id=XC_5Gh6y3wK},
}
Figure