Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap

Yongwei Chen, Zihao Wang, Longkun Zou, Ke Chen, Kui Jia

TL;DR

This paper tackles the "simulation-to-reality" (Sim2Real) gap in 3D object point cloud analysis by introducing a novel method that combines physically realistic synthesis of noisy point clouds with a quasi-balanced self-training strategy. This approach significantly improves the performance of point cloud classification models on real-world data by better mimicking the imperfections found in sensor captures.

What Problem Does This Paper Address?

Deep learning models for 3D object analysis, such as shape classification and part segmentation, heavily rely on large datasets. While synthetic datasets generated from CAD models are abundant, they often lack the realistic noise, incompleteness, and non-uniform density present in point clouds captured by real-world sensors. This "Sim2Real" domain gap leads to a significant drop in performance when models trained on synthetic data are applied to real-world scenarios. Existing domain adaptation techniques often struggle to bridge this gap effectively, especially when dealing with the complex noise patterns inherent in 3D sensing.

Methods

The paper proposes an integrated scheme called Quasi-Balanced Self-Training on Speckle-projected Synthesis (QS³). This scheme has two main components:

  1. Physically Realistic Synthesis (Mesh-to-Point): Instead of simple noise addition, the authors generate synthetic point clouds by simulating a realistic depth sensor. This involves:
  2. Using photorealistic rendering (Blender) of CAD models.
  3. Projecting speckle patterns onto the models, mimicking active stereo depth sensors.
  4. Simulating systematic sensor noises (e.g., from projection patterns and depth computation) to capture realistic shape and density variations.
  5. This process generates a new dataset called SpeckleNet.
  6. Quasi-Balanced Self-Training (QBST): To further bridge the Sim2Real gap and handle potential class imbalance in the target domain, a novel self-training strategy is introduced. QBST focuses on:
  7. Target Instance Selection: It dynamically selects unlabeled target samples for pseudo-labeling. This selection is guided by a "sparsity-driven weight" that prioritizes samples from under-represented (long-tailed) classes, aiming for a more balanced pseudo-labeled dataset.
  8. Self-Training: The model is then trained using these quasi-balanced pseudo-labeled target samples, alongside the original labeled source data. This process iteratively refines the model's ability to generalize to the target domain.

Datasets

  • SpeckleNet: A new synthetic dataset introduced by the authors, generated using their Mesh-to-Point pipeline. It's designed to simulate realistic noise and density variations found in real-world depth sensors.
  • ModelNet10: A standard synthetic dataset used for comparison and as a source domain in some experiments.
  • DepthScanNet10: A real-world dataset created by the authors from the ScanNet dataset. It's specifically designed to be more challenging than existing benchmarks by using raw depth sensor outputs without meshing, thus preserving realistic noise.
  • ScanNet10: Used as the basis for creating DepthScanNet10.

Key Insights and Ideas

  • Realistic Noise Simulation is Crucial: The paper strongly argues that simply adding random noise is insufficient. Simulating the physical process of depth sensing, including projection patterns and depth computation, is key to capturing the complex, non-uniform noises that truly affect real-world point clouds.
  • SpeckleNet Captures Realistic Imperfections: The SpeckleNet dataset, generated through their physically-based rendering, demonstrates superior performance when used as a source domain compared to standard synthetic datasets like ModelNet10, highlighting its effectiveness in bridging the Sim2Real gap.
  • Class Balance in Self-Training is Tricky: Standard self-training can be biased by easy classes. The proposed QBST addresses this by using a sparsity-aware selection mechanism to create a more balanced set of pseudo-labeled target samples, preventing negative transfer and improving robustness for long-tailed classes.
  • Integrated Approach is Powerful: Combining realistic data synthesis with a carefully designed self-training strategy (QS³) yields state-of-the-art results, suggesting that addressing both input-level (data synthesis) and training-level (self-training) challenges is essential for robust Sim2Real adaptation.

Key Results

  • The proposed QS³ scheme achieved state-of-the-art performance on the challenging DepthScanNet10 dataset, reaching an accuracy of 72.4%.
  • Training on the SpeckleNet synthetic data consistently outperformed training on ModelNet10 when tested on real data (DepthScanNet), demonstrating the effectiveness of their realistic synthesis approach. For example, ordinary classifiers trained on SpeckleNet achieved significantly higher accuracy (S→D column in Table 1) than when trained on ModelNet (M→D column).
  • The QBST component, when used with a DGCNN backbone and PCM augmentation, also showed superior performance compared to other self-training methods like CBST and SPST, especially in scenarios with imbalanced data.

Limitations and Future Work

  • Non-Differentiable Synthesis: The Mesh-to-Point synthesis pipeline is currently not end-to-end differentiable, meaning parameter settings are empirically chosen and might not be optimal for all tasks.
  • Future Work: The authors suggest incorporating differentiable physics-based rendering into their QS³ scheme for end-to-end optimization, which could lead to even better generation and adaptation. They also hint at exploring other sensor simulation techniques beyond speckle patterns.

Summary updated

BibTeX
@inproceedings{chen2022quasi,
  author = {Yongwei Chen and Zihao Wang and Longkun Zou and Ke Chen and Kui Jia},
  title = {{Quasi-Balanced Self-Training on Noise-Aware Synthesis of Object Point Clouds for Closing Domain Gap}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19827-4\_42},
}
Figure