Joint Supervised and Self-Supervised Learning for 3D Real World Challenges

Antonio Alliegro, Davide Boscaini, Tatiana Tommasi

Introduction

Deep learning has achieved remarkable success in 2D image processing, yet 3D point clouds—data typically captured by LiDAR or depth cameras—present unique challenges. Point clouds are inherently unstructured and unordered, making human annotation both difficult and time-consuming. This paper addresses three primary obstacles in 3D understanding: data scarcity in niche applications, the domain gap between synthetic CAD models and noisy real-world scans, and the difficulty models face in grasping global spatial relationships.

Methods: The 3D Puzzle-Solving Framework

The authors propose a multi-task learning framework that integrates a 3D Puzzle-Solving pretext task. This auxiliary task is designed to force the model to learn global structural features alongside its primary objective.

Overview of the 3D puzzle framework and real-world challenges.

The 3D Puzzle Task

A point cloud is first scaled into a unit cube and divided into an $l \times l \times l$ grid of voxels (typically $3 \times 3 \times 3$). Each point is labeled according to its original voxel. After randomly shuffling these voxels, the model must predict the original voxel index for every point.

Multi-Task Architecture

The framework utilizes a shared Feature Encoder (compatible with PointNet or PointNet++ backbones) that feeds into two distinct heads:

  1. Main Task Head: Handles standard classification or segmentation.
  2. Puzzle Head: Solves the spatial reordering problem.

By training on both tasks simultaneously, the encoder develops a representation that balances fine-grained geometric details with global spatial context.

Datasets and Evaluation

The approach was validated across several benchmarks:

  • Synthetic Data: ModelNet40 (12k CAD models) and ShapeNet (51k models).
  • Real-World Data: ScanObjectNN (2,902 scans), characterized by sensor noise and occlusions.
  • Cross-Domain Tasks: Evaluation of Domain Generalization (DG) and Domain Adaptation (DA) by training on synthetic data and testing on real-world scans.

Key Insights

  • Spatial Co-location: Solving the puzzle provides "spatial common sense," helping the network recognize objects despite noise or missing parts.
  • Self-Supervision as a Regularizer: In few-shot scenarios, the puzzle task prevents overfitting by forcing the model to learn meaningful geometric structures rather than memorizing small datasets.
  • Bridging the Domain Gap: The geometric logic of a shape remains consistent across synthetic and real domains, allowing the model to focus on underlying structure rather than domain-specific noise.

Results

The proposed method achieves state-of-the-art performance, particularly in domain adaptation and noisy environments.

Performance comparison on ModelNet40 and ScanObjectNN datasets.
  • Domain Adaptation: On the ModelNet40 $\rightarrow$ ScanObjectNN transfer task, the method outperformed previous leaders like PointDAN by up to 8 percentage points.
  • Robustness: In challenging ScanObjectNN variants with background clutter, accuracy improved by ~3% over standard baselines.
  • Few-Shot Learning: With only 20% labeled data, the self-supervised task significantly bolstered accuracy, demonstrating the utility of learning from unlabeled structural data.

Limitations and Future Work

While effective, the method's performance is sensitive to hyperparameters like the puzzle loss weight and grid size. The current rigid voxel grid could be replaced by more adaptive decomposition methods in future research. Additionally, exploring other pretext tasks, such as rotation prediction or point cloud completion, may further enhance 3D spatial understanding.

Summary updated

Abstract

Point cloud processing and 3D shape understanding are challenging tasks for which deep learning techniques have demonstrated great potentials. Still further progresses are essential to allow artificial intelligent agents to interact with the real world. In many practical conditions the amount of annotated data may be limited and integrating new sources of knowledge becomes crucial to support autonomous learning. Here we consider several scenarios involving synthetic and real world point clouds where supervised learning fails due to data scarcity and large domain gaps. We propose to enrich standard feature representations by leveraging self-supervision through a multi-task model that can solve a 3D puzzle while learning the main task of shape classification or part segmentation. An extensive analysis investigating few-shot, transfer learning and cross-domain settings shows the effectiveness of our approach with state-of-the-art results.
BibTeX
@inproceedings{alliegro2020joint,
  author = {Antonio Alliegro and Davide Boscaini and Tatiana Tommasi},
  title = {{Joint Supervised and Self-Supervised Learning for 3D Real World Challenges}},
  booktitle = {ICPR},
  year = {2020},
  url = {https://openreview.net/forum?id=rXlKQrIZQiw},
}
Figure