SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data

Mohammad Zohaib, Alessio Del Bue

Abstract

This paper proposes a new method to infer keypoints from arbitrary object categories in practical scenarios where point cloud data (PCD) are noisy, down-sampled and arbitrarily rotated. Our proposed model adheres to the following principles: i) keypoints inference is fully unsupervised (no annotation given), ii) keypoints position error should be low and resilient to PCD perturbations (robustness), iii) keypoints should not change their indexes for the intra-class objects (semantic coherence), iv) keypoints should be close to or proximal to PCD surface (compactness). We achieve these desiderata by proposing a new self-supervised training strategy for keypoints estimation that does not assume any a priori knowledge of the object class, and a model architecture with coupled auxiliary losses that promotes the desired keypoints properties. The experiments show that our approach outperforms state-of-the-art unsupervised approaches by estimating keypoints with improved coverage (+9.41%) while being semantically consistent (+4.66%).

Introduction

In 3D computer vision, keypoints are sparse coordinates defining an object's shape and pose, essential for tasks like tracking and robot manipulation. However, manual annotation is costly and prone to error. Furthermore, real-world sensors often produce "messy" data—noisy, low-resolution, and arbitrarily rotated. Existing unsupervised methods frequently struggle with these perturbations, often failing to maintain semantic coherence (e.g., identifying the "left wing" consistently across different airplane models). SC3K addresses these challenges with a fully unsupervised, rotation-invariant, and robust framework.

Methodology

The SC3K framework utilizes a "PCD to 3D keypoints" network featuring a PointNet backbone and residual blocks. The core innovation is its self-supervised training paradigm.

The SC3K self-supervised training paradigm and network architecture.

Self-Supervised Training Paradigm

The system employs a Siamese approach: a single point cloud is transformed into two versions via random rotations. The network estimates keypoints for both. A Mutual Dependency Loss penalizes misalignment when keypoints are "un-rotated" to a canonical pose. Additionally, a Pose Loss requires the network to estimate the relative rotation between the two sets, serving as a proxy task to ensure consistent keypoint placement.

Geometric Auxiliary Losses

To ensure keypoints are well-behaved, the model incorporates several auxiliary losses:

  • Separation & Overlap: Prevents keypoints from clustering or occupying the same coordinates.
  • Shape: Constrains keypoints to remain near the object's surface.
  • Volume: Ensures the keypoint-defined volume matches the original object's volume.

Experiments and Results

Evaluation was conducted on the KeypointNet dataset, which contains over 8,000 objects across 16 categories. While ground-truth labels exist, they were only used for validation.

Performance and Robustness

SC3K achieved state-of-the-art results, outperforming previous methods like ULCS and Skeleton Merger. It showed a +9.41% improvement in coverage and a +4.66% increase in semantic consistency (Dual Alignment Score).

Robustness of SC3K to noise and down-sampling.

A key strength is its resilience to data degradation. As shown in the robustness analysis, keypoint stability is maintained even under significant noise and extreme decimation (up to 32x down-sampling, where 97% of points are removed).

Conclusion and Limitations

SC3K demonstrates that rotation-invariant, semantically coherent keypoints can be learned without human supervision. However, limitations remain: performance degrades beyond 35 keypoints as they may fall into "empty gaps" in complex shapes. Symmetrical objects (like cars) can cause orientation confusion, and extreme intra-class variation (e.g., bunk beds vs. single beds) still poses a challenge for perfect alignment. Future work will explore higher-resolution keypoint sets and symmetry-aware loss functions.

Summary updated

Abstract

This paper proposes a new method to infer keypoints from arbitrary object categories in practical scenarios where point cloud data (PCD) are noisy, down-sampled and arbitrarily rotated. Our proposed model adheres to the following principles: i) keypoints inference is fully unsupervised (no annotation given), ii) keypoints position error should be low and resilient to PCD perturbations (robustness), iii) keypoints should not change their indexes for the intra-class objects (semantic coherence), iv) keypoints should be close to or proximal to PCD surface (compactness). We achieve these desiderata by proposing a new self-supervised training strategy for keypoints estimation that does not assume any a priori knowledge of the object class, and a model architecture with coupled auxiliary losses that promotes the desired keypoints properties. We compare the keypoints estimated by the proposed approach with those of the state-of-the-art unsupervised approaches. The experiments show that our approach outperforms by estimating keypoints with improved coverage (+9.41%) while being semantically consistent (+4.66%) that best characterize the object’s 3D shape for downstream tasks. Code and data are available at: https://github.com/IIT-PAVIS/SC3K
BibTeX
@inproceedings{zohaib2023sc,
  author = {Mohammad Zohaib and Alessio Del Bue},
  title = {{SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=JYpAwbLt0o},
}
Figure