Registration of Human Point Set Using Automatic Key Point Detection and Region-Aware Features

Amar Maharjan, Xiaohui YuanWACV 2022

TL;DR

This paper introduces a novel non-rigid point set registration method that automatically detects key points and uses region-aware features to establish correspondences, even with significant deformations and varying point densities. This approach significantly improves registration accuracy for complex human poses compared to existing methods, approaching the performance of those relying on manual key point annotations.

What Problem Does This Paper Address?

Registering 3D point sets, especially those representing human bodies in motion, is a challenging task due to large deformations and differences in the number of points captured by various sensors. Traditional methods often rely on manually defined "key points" (like joints) or feature descriptors that struggle with noise, incomplete data, and varying point densities. Manually annotating key points is time-consuming and impractical for large datasets. Existing automatic methods for finding key point correspondences often fail when point sets have different resolutions or are noisy, leading to inaccurate registrations. This paper aims to bridge this gap by providing a robust, automatic method for non-rigid registration that handles these challenges effectively.

Methods

The proposed method combines two key constraints: key point correspondences and local neighborhood preservation.

  1. Key Point Detection: Key points are automatically detected by identifying geodesic extrema within the point set. This leverages the concept of geodesic distance on the surface represented by the point cloud.
  2. Region-Aware Feature Descriptor: A novel feature descriptor is introduced to establish correspondences between these detected key points. Instead of relying on local neighborhood structures (which can be unreliable with varying densities), this descriptor uses statistics from clusters of points around each key point. Crucially, it encodes information about whether a cluster belongs to the "left-right" (symmetry) or "upper-lower" regions of the point set. This is achieved by analyzing geodesic distances from the cluster mean to specific points relative to the mean.
  3. Correspondence Establishment: An initial set of key point correspondences is computed using the proposed feature descriptors. These correspondences are then pruned to remove incorrect ones based on normalized weight values.
  4. Non-Rigid Registration: The registration itself is framed as a density estimation problem using Gaussian Mixture Models (GMMs). Two regularization terms are incorporated:
  5. Local Neighborhood Preservation: Stochastic Neighbor Embedding (SNE) is used to ensure that points that are close in the original point set remain close after transformation.
  6. Global Structure Constraint: This constraint uses the established key point correspondences to maintain the overall structure of the point set.
  7. An Expectation-Maximization (EM) algorithm is used to optimize the transformation.

Datasets

The paper uses a human dataset captured by Microsoft Kinect II, featuring various poses like stretching, squatting, and standing. Each point set contains approximately 12,000 points. For generalization experiments, they also use knife point sets from the Tools 2D database and cat point sets from the Nonrigid world 3D database.

Key Insights and Ideas

  • Region-Awareness is Key: The core innovation is the region-aware feature descriptor. By encoding left-right and upper-lower information within clusters, the method becomes more robust to noise and varying point densities than traditional local feature descriptors.
  • Geodesic Distance for Key Points: Using geodesic extrema for key point detection is a principled way to identify salient points on a non-rigid surface, especially when explicit anatomical landmarks are not available.
  • Balancing Global and Local Constraints: The paper demonstrates that both global (key point correspondences) and local (neighborhood preservation) constraints are crucial for accurate non-rigid registration, with global constraints showing a greater influence.
  • Robustness to Point Density: The method's ability to handle significant differences in the number of points between input and template sets is a major advantage for real-world applications.

Key Results

  • The proposed method achieves significantly lower registration errors compared to state-of-the-art methods like CPD, PR-GLS, and BCPD across various deformation levels.
  • For large deformations, the method's average registration error is only 3.9% higher than a method using manually specified key points, demonstrating its effectiveness in challenging scenarios.
  • The method shows good robustness to different numbers of points, with errors decreasing as the number of points in one set decreases.
  • While the method shows promise for non-human point sets like tools, it struggles with complex animal shapes (cats) where establishing key point correspondences is difficult, suggesting a need for further improvements in this area.

Limitations and Future Work

  • Non-Human Point Sets: The method's performance on non-human point sets, particularly complex organic shapes like animals, is limited by its reliance on establishing key point correspondences. Future work aims to improve this capability.
  • Manual vs. Automatic: While the method significantly closes the gap, it still exhibits a slight increase in error (3.9%) compared to methods using manually defined key points. Further refinement could potentially match or exceed manual annotation performance.
  • Computational Cost: Not explicitly detailed, but complex GMM optimization and feature computation can be computationally intensive.
  • Future Work: The authors explicitly mention extending the method to handle animal point sets and correctly establish key point correspondences for such data.

Summary updated

BibTeX
@inproceedings{maharjan2022registration,
  author = {Amar Maharjan and Xiaohui Yuan},
  title = {{Registration of Human Point Set Using Automatic Key Point Detection and Region-Aware Features}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure