Generalizable Single-View Object Pose Estimation by Two-Side Generating and Matching

Yujing Sun, Caiyi Sun, Yuan Liu, Yuexin Ma, Siu Ming Yiu

Abstract

In this paper, we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data, our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably, our approach maintains strong performance even in scenarios with significant viewpoint changes, highlighting its robustness and versatility in challenging conditions. The code is available at https://github.com/scy639/Gen2SM.

Introduction

Estimating the 3D orientation (pose) of an object from a single RGB image is a cornerstone of computer vision, essential for robotics and augmented reality. However, traditional methods usually fall into two traps: instance-level dependency, requiring detailed 3D CAD models, and category bias, where models fail when encountering novel objects outside their training set.

Recent "category-agnostic" methods attempt to solve this by using diffusion models to imagine an object from new angles. However, these models face a significant "bottleneck": as the viewpoint shift increases (e.g., beyond 120°), image quality degrades sharply, making it nearly impossible to match generated views with a real query image. Gen2SM addresses this challenge of large viewpoint changes by leveraging a diffusion model to "meet in the middle."

Methodology: Two-Side Generating and Matching

The core innovation of Gen2SM is the Two-side Generating and Matching strategy. Instead of attempting a direct jump from a reference image ($I_r$) to a query image ($I_q$), the framework generates a set of images at intermediate viewpoints from both sides.

The "meeting in the middle" intuition: (a) traditional one-side generation fails at large angles, while (b) two-side generation maintains quality by using smaller relative rotations.

The "Meeting in the Middle" Intuition

If the viewpoint difference between the reference and query is 120°, a single 120° rotation is difficult for current diffusion models to generate accurately. By generating a 60° rotation from the reference and a -60° rotation from the query, the two resulting images "meet" at the same intermediate viewpoint. These smaller generations are of significantly higher quality and much easier to match.

Generative Backbone and Loss Function

Gen2SM utilizes Zero123, a diffusion model fine-tuned for viewpoint conditioning. To compare images, the authors adapt the Score Distillation Sampling (SDS) loss. Rather than comparing pixels directly—which are prone to AI artifacts—the method compares the "noise residuals" predicted by the diffusion model. This internal denoising mechanism acts as a robust similarity sensor, outperforming standard feature matching on smooth or featureless AI-generated surfaces.

Optimization

The pose estimation follows a coarse-to-fine approach:

  1. Grid-based search: A global search over a sphere of possible angles to find a rough estimate.
  2. Gradient Descent: Refinement of the pose to achieve higher precision using the matching loss.

Experiments and Results

The researchers evaluated Gen2SM on two primary benchmarks: the synthetic GSO (Google Scanned Objects) dataset and the real-world NAVI dataset, which features images captured in diverse environments with varying lighting and backgrounds.

Qualitative results showing estimated poses (red/green boxes) for various objects in the GSO and NAVI datasets.

Key Performance Metrics

Gen2SM significantly outperformed existing state-of-the-art (SOTA) methods, particularly in difficult scenarios:

  • Overall Accuracy: On the real-world NAVI dataset, Gen2SM achieved 55.32% accuracy (within 15° error), compared to the previous best diffusion-based method (E2VG) at 43.16%.
  • Large Viewpoint Robustness: In cases where the viewpoint change exceeded 150°, Gen2SM maintained 21.82% accuracy, while competitors like 3DAHV and IDPose struggled at 14.55% and 9.09% respectively.
  • Qualitative Success: Visual results demonstrate the method successfully handling complex objects like toys, kettles, and furniture where traditional methods often fail due to lack of distinct textures.

Discussion and Limitations

The success of Gen2SM proves that we do not necessarily need to reconstruct a 3D mesh to understand 3D orientation; the geometric priors inside large-scale diffusion models contain enough understanding to act as a proxy. However, some limitations remain:

  • Inference Speed: The optimization process is computationally intensive, taking several seconds per object on an A100 GPU. While a "light" version exists, it is not yet ready for real-time robotics.
  • Model Dependency: Accuracy is capped by the quality of the underlying diffusion model (Zero123). Improvements in base models will directly translate to performance boosts for Gen2SM.
  • Object-Centricity: The method currently requires the object to be somewhat centered and cropped, and it does not yet handle complex, cluttered scenes with significant overlaps.

Summary updated

Abstract

In this paper, we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data, our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably, our approach maintains strong performance even in scenarios with significant viewpoint changes, highlighting its robustness and versatility in challenging conditions. The code will be released at https://github.com/scy639/Gen2SM.
BibTeX
@inproceedings{sun2025generalizable,
  author = {Yujing Sun and Caiyi Sun and Yuan Liu and Yuexin Ma and Siu Ming Yiu},
  title = {{Generalizable Single-View Object Pose Estimation by Two-Side Generating and Matching}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=vW8Svv6Ysf},
}
Figure