Abstract
This paper proposes a new method to infer keypoints from arbitrary object categories in practical scenarios where point cloud data (PCD) are noisy, down-sampled and arbitrarily rotated. Our proposed model adheres to the following principles: i) keypoints inference is fully unsupervised (no annotation given), ii) keypoints position error should be low and resilient to PCD perturbations (robustness), iii) keypoints should not change their indexes for the intra-class objects (semantic coherence), iv) keypoints should be close to or proximal to PCD surface (compactness). We achieve these desiderata by proposing a new self-supervised training strategy for keypoints estimation that does not assume any a priori knowledge of the object class, and a model architecture with coupled auxiliary losses that promotes the desired keypoints properties. The experiments show that our approach outperforms state-of-the-art unsupervised approaches by estimating keypoints with improved coverage (+9.41%) while being semantically consistent (+4.66%).
Introduction
In 3D computer vision, keypoints are sparse coordinates defining an object's shape and pose, essential for tasks like tracking and robot manipulation. However, manual annotation is costly and prone to error. Furthermore, real-world sensors often produce "messy" data—noisy, low-resolution, and arbitrarily rotated. Existing unsupervised methods frequently struggle with these perturbations, often failing to maintain semantic coherence (e.g., identifying the "left wing" consistently across different airplane models). SC3K addresses these challenges with a fully unsupervised, rotation-invariant, and robust framework.
Methodology
The SC3K framework utilizes a "PCD to 3D keypoints" network featuring a PointNet backbone and residual blocks. The core innovation is its self-supervised training paradigm.
Self-Supervised Training Paradigm
The system employs a Siamese approach: a single point cloud is transformed into two versions via random rotations. The network estimates keypoints for both. A Mutual Dependency Loss penalizes misalignment when keypoints are "un-rotated" to a canonical pose. Additionally, a Pose Loss requires the network to estimate the relative rotation between the two sets, serving as a proxy task to ensure consistent keypoint placement.
Geometric Auxiliary Losses
To ensure keypoints are well-behaved, the model incorporates several auxiliary losses:
- Separation & Overlap: Prevents keypoints from clustering or occupying the same coordinates.
- Shape: Constrains keypoints to remain near the object's surface.
- Volume: Ensures the keypoint-defined volume matches the original object's volume.
Experiments and Results
Evaluation was conducted on the KeypointNet dataset, which contains over 8,000 objects across 16 categories. While ground-truth labels exist, they were only used for validation.
Performance and Robustness
SC3K achieved state-of-the-art results, outperforming previous methods like ULCS and Skeleton Merger. It showed a +9.41% improvement in coverage and a +4.66% increase in semantic consistency (Dual Alignment Score).
A key strength is its resilience to data degradation. As shown in the robustness analysis, keypoint stability is maintained even under significant noise and extreme decimation (up to 32x down-sampling, where 97% of points are removed).
Conclusion and Limitations
SC3K demonstrates that rotation-invariant, semantically coherent keypoints can be learned without human supervision. However, limitations remain: performance degrades beyond 35 keypoints as they may fall into "empty gaps" in complex shapes. Symmetrical objects (like cars) can cause orientation confusion, and extreme intra-class variation (e.g., bunk beds vs. single beds) still poses a challenge for perfect alignment. Future work will explore higher-resolution keypoint sets and symmetry-aware loss functions.
Summary updated