CatFormer: Category-Level 6D Object Pose Estimation with Transformer

Sheng Yu, Di-Hua Zhai, Yuanqing XiaAAAI 2024 · DOI

Introduction

Traditional 6D pose estimation—determining an object's 3D rotation and translation—typically relies on instance-level models, requiring precise CAD models for every target object. This approach fails when encountering novel objects within a known category. Category-level estimation addresses this by generalizing across objects like "cameras" or "laptops," which vary significantly in shape and size. However, this task is complicated by intra-class variety, fragmented sensor data due to occlusions, and the limitations of static deformation models that attempt to predict shape in a single pass.

Methodology

CatFormer introduces a Transformer-centric architecture designed to adapt a generic category "prior" (e.g., a template mug) into a specific, accurate 3D model of a target object. The framework operates through a three-stage pipeline:

  1. Coarse Deformation: This module processes incomplete point clouds from depth sensors, using a transformer-based module to "fill in" missing geometry and create a more complete representation.
  2. Fine Deformation: Starting with a category-specific "prior point cloud," the system uses cross-attention to fuse features from the RGB image and the scene point cloud, deforming the template to match the target's specific geometry.
  3. Recurrent Refinement: Rather than a single prediction, the model iteratively updates the deformation and transformation matrix. The authors found that five iterations allow the prior point cloud to gradually "wrap" tightly around the actual object, significantly increasing precision.
The three-stage pipeline of CatFormer: Coarse Deformation, Fine Deformation, and Recurrent Refinement.

A key innovation is the Transformer-based Graph Module, which establishes geometric and topological relationships between points. By treating points as nodes in a graph and applying attention mechanisms, the model captures both local neighborhoods and the overall object structure.

Experiments and Results

The authors evaluated CatFormer on major benchmarks including CAMERA25, REAL275, and LINEMOD. On the challenging REAL275 real-world dataset, CatFormer achieved state-of-the-art (SOTA) results:

  • Accuracy: Scored 83.1 on the $IoU_{50}$ metric, surpassing the previous SOTA (HS-Pose) of 82.1.
  • Precision: Reached 47.7 on the strict $5^\circ 2cm$ metric, representing a significant jump over previous methods.
  • Generalization: The model successfully estimated poses for unseen real-world objects and outperformed specialized instance-level methods on LINEMOD with an ADD-(S) score of 99.3.
Qualitative comparison of 6D pose estimation results between SGPA and CatFormer on the REAL275 dataset.

Conclusion

CatFormer demonstrates that Transformers are highly effective for 3D geometry tasks. By combining graph-based reasoning with iterative refinement, the model achieves high precision even with fragmented data. While the system currently depends on initial 2D masks from Mask-RCNN and has a computational overhead (running at ~8 FPS), it sets a new standard for category-level pose estimation. Future work may explore reducing category-specific priors toward zero-shot estimation.

Summary updated

BibTeX
@inproceedings{yu2024catformer,
  author = {Sheng Yu and Di-Hua Zhai and Yuanqing Xia},
  title = {{CatFormer: Category-Level 6D Object Pose Estimation with Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28505},
}
Figure