Abstract
In this paper, we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data, our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably, our approach maintains strong performance even in scenarios with significant viewpoint changes, highlighting its robustness and versatility in challenging conditions. The code is available at https://github.com/scy639/Gen2SM.
Introduction
Estimating the 3D orientation (pose) of an object from a single RGB image is a cornerstone of computer vision, essential for robotics and augmented reality. However, traditional methods usually fall into two traps: instance-level dependency, requiring detailed 3D CAD models, and category bias, where models fail when encountering novel objects outside their training set.
Recent "category-agnostic" methods attempt to solve this by using diffusion models to imagine an object from new angles. However, these models face a significant "bottleneck": as the viewpoint shift increases (e.g., beyond 120°), image quality degrades sharply, making it nearly impossible to match generated views with a real query image. Gen2SM addresses this challenge of large viewpoint changes by leveraging a diffusion model to "meet in the middle."
Methodology: Two-Side Generating and Matching
The core innovation of Gen2SM is the Two-side Generating and Matching strategy. Instead of attempting a direct jump from a reference image ($I_r$) to a query image ($I_q$), the framework generates a set of images at intermediate viewpoints from both sides.
The "Meeting in the Middle" Intuition
If the viewpoint difference between the reference and query is 120°, a single 120° rotation is difficult for current diffusion models to generate accurately. By generating a 60° rotation from the reference and a -60° rotation from the query, the two resulting images "meet" at the same intermediate viewpoint. These smaller generations are of significantly higher quality and much easier to match.
Generative Backbone and Loss Function
Gen2SM utilizes Zero123, a diffusion model fine-tuned for viewpoint conditioning. To compare images, the authors adapt the Score Distillation Sampling (SDS) loss. Rather than comparing pixels directly—which are prone to AI artifacts—the method compares the "noise residuals" predicted by the diffusion model. This internal denoising mechanism acts as a robust similarity sensor, outperforming standard feature matching on smooth or featureless AI-generated surfaces.
Optimization
The pose estimation follows a coarse-to-fine approach:
- Grid-based search: A global search over a sphere of possible angles to find a rough estimate.
- Gradient Descent: Refinement of the pose to achieve higher precision using the matching loss.
Experiments and Results
The researchers evaluated Gen2SM on two primary benchmarks: the synthetic GSO (Google Scanned Objects) dataset and the real-world NAVI dataset, which features images captured in diverse environments with varying lighting and backgrounds.
Key Performance Metrics
Gen2SM significantly outperformed existing state-of-the-art (SOTA) methods, particularly in difficult scenarios:
- Overall Accuracy: On the real-world NAVI dataset, Gen2SM achieved 55.32% accuracy (within 15° error), compared to the previous best diffusion-based method (E2VG) at 43.16%.
- Large Viewpoint Robustness: In cases where the viewpoint change exceeded 150°, Gen2SM maintained 21.82% accuracy, while competitors like 3DAHV and IDPose struggled at 14.55% and 9.09% respectively.
- Qualitative Success: Visual results demonstrate the method successfully handling complex objects like toys, kettles, and furniture where traditional methods often fail due to lack of distinct textures.
Discussion and Limitations
The success of Gen2SM proves that we do not necessarily need to reconstruct a 3D mesh to understand 3D orientation; the geometric priors inside large-scale diffusion models contain enough understanding to act as a proxy. However, some limitations remain:
- Inference Speed: The optimization process is computationally intensive, taking several seconds per object on an A100 GPU. While a "light" version exists, it is not yet ready for real-time robotics.
- Model Dependency: Accuracy is capped by the quality of the underlying diffusion model (Zero123). Improvements in base models will directly translate to performance boosts for Gen2SM.
- Object-Centricity: The method currently requires the object to be somewhat centered and cropped, and it does not yet handle complex, cluttered scenes with significant overlaps.
Summary updated