Instance-Adaptive Keypoint Learning with Local-to-Global Geometric Aggregation for Category-Level Object Pose Estimation

Xiao Zhang, Lu Zou, Tao Lu, Yuan Yao, Zhangjin Huang, Guoping WangarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces INKL-Pose, a novel framework for category-level 6D object pose estimation that excels at handling complex object shapes and variations. It achieves state-of-the-art results by adaptively learning keypoints and aggregating their features both locally and globally using a bidirectional Mamba architecture.

What Problem Does This Paper Address?

Estimating the 6D pose (position and orientation) and size of objects from images is crucial for robotics and computer vision. While "instance-level" methods work well for known objects with precise 3D models, they fail when encountering new instances. "Category-level" pose estimation aims to solve this by predicting pose for unseen objects within a category, but it's significantly harder due to variations in shape, texture, and structure within a category. Existing methods often struggle with objects that deviate from typical shapes or have complex geometries, leading to inaccurate keypoint predictions and ultimately poor pose estimation.

Methods

INKL-Pose tackles this challenge with a multi-stage approach:

  1. Instance-Adaptive Keypoint Detector (IAKD): This module, adapted from prior work, dynamically generates semantically consistent and geometrically informative keypoints for each object instance. It avoids fixed sampling strategies by learning to identify the most informative regions.
  2. Local Keypoint Feature Aggregator (LKFA): To capture fine-grained geometric details, this module enriches each keypoint's features by aggregating information from its local neighborhood. It uses a K-Nearest Neighbors (KNN) approach and positional encoding to understand local structures.
  3. Global Keypoint Feature Aggregator (GKFA): This is a core innovation. It uses a bidirectional Mamba architecture to model long-range dependencies and capture global structural consistency across the object. To handle the unordered nature of point clouds and enable bidirectional processing, they introduce a Feature Sequence Flipping (FSF) strategy. This strategy reverses feature channels within keypoint vectors, allowing for richer feature interactions without disrupting spatial arrangements.
  4. Surface and Separation Losses: To ensure keypoints are well-distributed and diverse, they introduce two novel loss functions:
  5. Surface Loss (Lsurf): Encourages uniform coverage of the object's surface.
  6. Separation Loss (Lsep): Promotes spatial diversity by ensuring keypoints are distinct from their local neighbors.
  7. Pose and Size Estimator: Finally, the refined keypoints and their features are used to regress the object's 6D pose and size.

Datasets

The paper evaluates INKL-Pose on three established datasets:

  • CAMERA25: A large-scale synthetic dataset.
  • REAL275: A real-world counterpart to CAMERA25.
  • HouseCat6D: A more challenging, real-world multimodal dataset featuring household objects, known for its complex scenes, clutter, and occlusions.

Key Insights and Ideas

  • Local-to-Global Aggregation is Key: The paper strongly advocates for combining fine-grained local geometric details with long-range structural understanding for robust pose estimation, especially for complex objects.
  • Mamba for Global Context: Bidirectional Mamba proves highly effective for modeling long-range dependencies in keypoint features, outperforming standard attention mechanisms in this context.
  • Feature Sequence Flipping (FSF) for Bidirectionality: A simple yet clever trick to enable bidirectional Mamba on point cloud data by reversing feature channels, preserving spatial coherence.
  • Targeted Keypoint Supervision: The proposed Lsurf and Lsep losses are crucial for guiding the network to produce well-distributed and informative keypoints, significantly improving performance over unsupervised or less effective supervision methods.
  • Adaptive Keypoints Matter: The instance-adaptive nature of the keypoint detector is essential for handling diverse object instances within a category.

Key Results

  • State-of-the-Art Performance: INKL-Pose achieves state-of-the-art results on REAL275, CAMERA25, and HouseCat6D across various metrics, including IoU and n°mcm (rotation and translation error).
  • Significant Gains: On REAL275, it shows improvements of up to 6.2% under the strict 5°2cm metric compared to prior methods like GCE-Pose and CleanPose. On HouseCat6D, it outperforms AG-Pose by significant margins, especially on symmetric objects with ambiguity like Bottles and Mugs.
  • Efficiency: The model has 16.7M parameters and achieves a fast inference speed of 36 FPS on an NVIDIA RTX 4090D GPU, making it suitable for real-time applications.
  • Ablation Studies: The number of keypoints (96), LKFA/GKFA modules (12), and neighborhood size (K=4) were found to be optimal. The proposed Lsurf and Lsep losses are critical for performance.

Limitations and Future Work

  • Static, Single-Frame: The current framework processes static, single-frame observations and doesn't leverage temporal information from video sequences, which could improve robustness in dynamic scenarios.
  • Challenging Materials: Performance can degrade with highly reflective or transparent objects due to difficulties in reliable semantic feature extraction.
  • Severe Occlusion & Atypical Geometries: Extreme self-occlusion and object shapes that deviate significantly from typical category patterns remain challenging.
  • Keypoint Localization Errors: Errors in keypoint detection can propagate to pose estimation.

Future work includes extending the framework to temporal settings for pose tracking, exploring more robust semantic feature extraction, and incorporating uncertainty-aware keypoint modeling.

Summary updated

Abstract

Category-level object pose estimation aims to predict the 6D pose and size of previously unseen instances from predefined categories, requiring strong generalization across diverse object instances. Although many previous methods attempt to mitigate intra-class variations, they often struggle with instances exhibiting complex geometries or significant deviations from canonical shapes. To address this issue, we propose INKL-Pose, a novel category-level object pose estimation framework that enables INstance-adaptive Keypoint Learning with local-to-global geometric aggregation. Specifically, our method first predicts semantically consistent and geometrically informative keypoints using an Instance-Adaptive Keypoint Detector, then refines them: (1) a Local Keypoint Feature Aggregator capturing fine-grained geometries, and (2) a Global Keypoint Feature Aggregator using bidirectional Mamba for structural consistency. To enable bidirectional modeling in Mamba, we introduce a simple yet effective Feature Sequence Flipping strategy that preserves spatial coherence while constructing backward feature sequence. Additionally, we design a surface loss and a separation loss to encourage uniform coverage and spatial diversity in keypoint distribution. The resulting keypoints are mapped to a canonical space for 6D pose and size regression. Extensive experiments on CAMERA25, REAL275, and HouseCat6D show that INKL-Pose achieves state-of-the-art performance with 16.7M parameters and runs at 36 FPS on an NVIDIA RTX 4090D GPU.
BibTeX
@misc{zhang2025instance,
  author = {Xiao Zhang and Lu Zou and Tao Lu and Yuan Yao and Zhangjin Huang and Guoping Wang},
  title = {{Instance-Adaptive Keypoint Learning with Local-to-Global Geometric Aggregation for Category-Level Object Pose Estimation}},
  year = {2025},
  eprint = {2504.15134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.15134},
}
Figure