Unsupervised Learning of Category-Specific Symmetric 3D Keypoints from Point Sets

Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel, José Jesús Guerrero, Cédric Demonceaux, Luc Van Gool

Abstract

This paper presents an unsupervised framework for discovering 3D keypoints directly from misaligned point clouds. By integrating low-rank non-rigid shape modeling with a symmetry prior, the proposed method learns semantically consistent and ordered keypoints across object categories, such as chairs and humans, without requiring manual annotations or pre-aligned data. The approach leverages a symmetric shape basis and a symmetric deformation space to handle both rigid and non-rigid objects, achieving high semantic consistency and superior object coverage compared to existing unsupervised methods.

Introduction

Identifying 3D keypoints—points representing the structural and semantic essence of an object—is vital for 3D reconstruction, registration, and recognition. Traditional methods often rely on expensive manual annotations or 2D image-based supervision. The challenge lies in performing this task unsupervised and directly on 3D point sets, which are typically unordered, misaligned, and exhibit significant intra-category shape variation. This work addresses the problem of "category-specificity," ensuring that keypoints correspond to the same semantic parts across different instances of a category.

Methodology

The framework treats keypoint discovery as a non-rigid modeling problem, inspired by Non-Rigid Structure-from-Motion (NRSfM). Objects are represented as linear combinations of a learned "shape basis."

  • Symmetric Shape Basis: To handle rigid objects like cars or chairs, the model incorporates reflective symmetry as a geometric prior. This reduces pose ambiguity and guides the network toward semantically meaningful locations.
  • Symmetric Deformation Space: For non-rigid objects like humans, where individual poses may not be symmetric, the authors propose a model where the collection of possible poses is symmetric.
Keypoint discovery on non-rigid human point clouds in different poses.

The architecture consists of a Node Branch (PointNet-based) for proposing keypoint locations and a Pose and Coefficients Branch for predicting rotation and basis weights. Training is guided by a combination of Chamfer, Coverage, and Inclusivity losses to ensure alignment, full-object representation, and surface-constrained keypoints.

Experiments and Datasets

The method was evaluated on four benchmarks to demonstrate its versatility across rigid and non-rigid categories:

  • ModelNet10: Indoor rigid objects (chairs, tables, beds).
  • ShapeNet Parts: Outdoor rigid objects (airplanes, cars, motorbikes).
  • Dynamic FAUST (D-FAUST): High-resolution 4D human scans in motion.
  • Basel Face Model 2017: Synthetic 3D faces with varying expressions.

Results and Discussion

The framework achieves approximately 93% semantic consistency. Keypoints are inherently ordered because they are generated from a fixed basis, enabling immediate point-to-point registration without additional matching steps.

Visualization of semantically consistent keypoints across various object categories and shapes.

Key findings include:

  • Robustness: The model successfully discovers keypoints on data with random rotations up to ±45 degrees.
  • Coverage: It provides 29.4% better volume coverage than the USIP baseline, ensuring keypoints represent the whole shape.
  • Symmetry as a Prior: Forcing a symmetric basis significantly aids convergence in the unsupervised search space, helping the network find semantically meaningful locations rather than just mathematically convenient ones.

Conclusion

While the linear basis model is effective, future work could explore non-linear generative models for highly articulated deformations. Additionally, extending the framework to handle multi-category training and significant part occlusions in real-world LiDAR data remains an open challenge.

Summary updated

BibTeX
@inproceedings{fernandezlabrador2020unsupervised,
  author = {Clara Fernandez-Labrador and Ajad Chhatkuli and Danda Pani Paudel and José Jesús Guerrero and Cédric Demonceaux and Luc Van Gool},
  title = {{Unsupervised Learning of Category-Specific Symmetric 3D Keypoints from Point Sets}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=8ZrPBwLjpt},
}
Figure