Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching

Yujing Sun, Caiyi Sun, Yuan Liu, Yuexin Ma, Siu Ming Yiu

Abstract

In this paper, we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data, our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably, our approach maintains strong performance even in scenarios with significant viewpoint changes, highlighting its robustness and versatility in challenging conditions.

Introduction

Determining the 3D orientation of an object from a single reference photograph is a foundational task for robotics, augmented reality, and autonomous driving. Traditional methods often struggle with generalization, requiring high-fidelity 3D CAD models or extensive training on specific object categories. This paper introduces Gen2SM, a "two-side" matching strategy that leverages diffusion models to bridge the gap between a reference image and a query image, enabling accurate pose estimation for never-before-seen objects.

The Generation Bottleneck

Recent "category-agnostic" methods attempt to use diffusion models, such as Zero123, to "hallucinate" an object's appearance from new angles. However, these models face a significant generation bottleneck: as the rotation angle increases (e.g., beyond 120 degrees), the quality of generated images degrades into blurry, distorted artifacts. Matching a crisp real-world query photo against these low-quality hallucinations leads to substantial estimation errors.

Gen2SM: Two-side Generating and Matching

To overcome the generation bottleneck, Gen2SM moves away from naive direct matching. Instead of attempting to generate the query view directly from the reference, the authors propose a Two-side Generation and Matching approach.

Comparison between naive matching (a) and the proposed two-side matching strategy (b).

The core idea is to "meet in the middle." The process involves:

  1. Intermediate Viewpoints: Sampling a set of $N$ viewpoints distributed evenly on a sphere around the object.
  2. The Two-Side Bridge: For a hypothesized pose, the system generates what the object should look like at intermediate viewpoints starting from the reference image.
  3. Score Distillation Matching: Rather than matching raw pixels, which are prone to generative artifacts, Gen2SM uses a technique inspired by DreamFusion. It adds noise to the generated intermediate images and utilizes the diffusion model—conditioned on the query image—to predict that noise.
  4. Optimization: The matching score is defined as the L2 distance between the added noise and the predicted noise. A coarse-to-fine search strategy, starting with a grid search followed by gradient descent, is used to refine the pose estimate.

By splitting a large viewpoint shift into two smaller, more manageable ones (Reference $\rightarrow$ Intermediate $\leftarrow$ Query), the generative model operates within its high-accuracy range, providing more reliable features for matching.

Evaluation and Results

The method was evaluated in a zero-shot manner on two primary benchmarks: GSO (Google Scanned Objects), a synthetic dataset of ~1,000 items, and NAVI, a challenging real-world dataset with precise ground-truth poses.

Qualitative results of Gen2SM on various query images with estimated poses.

Gen2SM significantly outperformed state-of-the-art (SOTA) baselines:

  • Real-World Accuracy: On the NAVI dataset, Gen2SM achieved 55.32% accuracy (within 15° error), a significant improvement over the 43.16% achieved by previous best methods.
  • Robustness to Large Shifts: For viewpoint changes exceeding 120°, Gen2SM maintained 31.71% accuracy, nearly 1.5x better than existing techniques that often fail in such extreme scenarios.
  • Efficiency: A "light" version of the model can process a query in approximately 1.12 seconds on an A100 GPU, making it practical for real-time applications.

Key Insights

  • Noise as a Universal Language: Matching in the "noise prediction space" of a diffusion model is more robust than pixel-space matching. The model's internal structural understanding acts as a powerful feature extractor.
  • Eliminating CAD Models: By leveraging the "world knowledge" of large-scale diffusion models, Gen2SM removes the requirement for 3D models or multiple reference views.

Limitations and Future Work

Despite its robustness, Gen2SM is an RGB-only approach and currently lacks the fine-grained precision (within 1-2 degrees) of methods using depth sensors. Additionally, because it relies on object-centric diffusion models, it may struggle in highly cluttered scenes with significant occlusions. Future improvements could involve integrating more advanced base models like SV3D to further enhance performance.

Summary updated

Abstract

In this paper, we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data, our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably, our approach maintains strong performance even in scenarios with significant viewpoint changes, highlighting its robustness and versatility in challenging conditions. The code will be re leased at https://github.com/scy639/Gen2SM.
BibTeX
@inproceedings{sun2025generalizable,
  author = {Yujing Sun and Caiyi Sun and Yuan Liu and Yuexin Ma and Siu Ming Yiu},
  title = {{Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=vW8Svv6Ysf},
}
Figure