Point-Syn2Real: Semi-Supervised Synthetic-to-Real Cross-Domain Learning for Object Classification in 3D Point Clouds

Ziwei Wang, Reza Arablouei, Jiajun Liu, Paulo Borges, Greg Bishop-Hurley, Nicholas Heaney

Abstract

Point-Syn2Real is a novel framework designed to train 3D object classifiers using readily available synthetic CAD models and unlabeled real-world data. By simulating realistic LiDAR occlusions and employing semi-supervised entropy minimization, this approach bypasses the need for labor-intensive manual point cloud annotation. Experimental results demonstrate state-of-the-art performance in transferring knowledge from simulation to reality across both indoor and outdoor datasets, nearly matching supervised performance in some cases.

Introduction

Deep learning models for 3D point clouds, critical for applications like autonomous driving and robotics, are notoriously data-hungry. However, labeling 3D data is a "labor-intensive nightmare"—human annotators must rotate and inspect sparse clusters of points from multiple angles to identify objects. While massive libraries of 3D CAD models exist, models trained on these "perfect" shapes often fail when deployed on real LiDAR sensors due to the significant domain gap. Real LiDAR scans are noisy, sparse, and capture only the surfaces facing the sensor (partial occlusion), whereas CAD models are complete and clean.

Comparison between traditional real-world data collection and the proposed Syn2Real cross-domain learning pipeline.

Methodology

The authors propose a three-pillar approach to bridge the synthetic-to-real gap without requiring labeled real-world data.

1. Realistic Multi-view Simulation

Instead of using complete 3D models, the framework uses simulation tools to replicate how a LiDAR sensor actually perceives an object. By placing a virtual sensor at random positions around a CAD model and back-projecting depth maps into point clouds, the system creates realistic, partially-occluded views. This ensures the model learns features that actually appear in real-world scans.

2. Robust Data Augmentation

To handle real-world variability, random z-axis rotations and Gaussian noise are applied to the synthetic samples. This teaches the model to be invariant to an object's orientation and sensor-induced imperfections.

3. Semi-Supervised Learning (Entropy Minimization)

The "secret sauce" of the framework is its training objective. The model is trained on labeled synthetic data using a weighted cross-entropy loss. Simultaneously, it processes unlabeled real-world data. The training objective encourages the model to make "confident" predictions on this unlabeled data by minimizing its Shannon entropy. This forces the model to refine its features to align with real-world distributions without needing explicit labels.

The Point-Syn2Real training framework, utilizing synthetic 3D models with labels and unlabeled real 3D scans for semi-supervised learning.

Experimental Evaluation

The authors introduced a new benchmark dataset, Point-Syn2Real, covering two distinct scenarios:

  • Indoor: ModelNet and ShapeNet (synthetic source) to ScanNet (real-world target).
  • Outdoor: A newly curated 3D_City subset (synthetic source) to SemanticKITTI (real-world target).

The proposed method significantly outperformed existing domain adaptation techniques. In the outdoor task, a baseline with no adaptation achieved only 19.76% accuracy. Point-Syn2Real reached 96.57% accuracy, nearly matching the performance of a fully supervised model (97.06%). On the indoor task, it achieved 59.13% accuracy, outperforming specialized methods like PointDAN (33.00%) and DefRec+PCM (51.80%).

Discussion and Conclusion

A key insight of this work is that simulating the physics of the sensor (partial views) is more critical than high-fidelity textures. By training on "broken" versions of CAD models, the model learns robust features. Furthermore, entropy minimization acts as an effective bridge, allowing the model to self-train on real-world data. Point-Syn2Real provides a scalable, backbone-agnostic solution to the 3D data bottleneck, making high-performance 3D classification accessible without the cost of manual annotation.

Summary updated

BibTeX
@inproceedings{wang2023point,
  author = {Ziwei Wang and Reza Arablouei and Jiajun Liu and Paulo Borges and Greg Bishop-Hurley and Nicholas Heaney},
  title = {{Point-Syn2Real: Semi-Supervised Synthetic-to-Real Cross-Domain Learning for Object Classification in 3D Point Clouds}},
  booktitle = {ICME},
  year = {2023},
  url = {https://openreview.net/forum?id=XC_5Gh6y3wK},
}
Figure