HIPPo: Harnessing Image-to-3D Priors for Model-Free Zero-Shot 6D Pose Estimation

Yibo Liu, Zhaodong Jiang, Binbin Xu, Guile Wu, Yuan Ren, Tongtong Cao, Bingbing Liu, Rui Heng Yang, Amir Rasouli, Jinjun Shan

Abstract

Estimating the 6D pose of unseen objects in the wild is a critical challenge for robotics. Traditional methods rely on pre-existing CAD models or extensive reference libraries, which are often unavailable in novel environments. We present HIPPo, a framework that leverages generative image-to-3D diffusion priors to enable model-free, zero-shot 6D pose estimation from a single RGB-D image. By "dreaming up" a complete 3D mesh and refining it with real-time observations, HIPPo bridges the gap between generative AI and precise robotic manipulation.

Introduction

In "open-world" robotics, such as a service robot entering a new home, the ability to interact with previously unseen objects is paramount. This requires knowing the object's 6D pose—its 3D position and orientation. Existing solutions typically require textured CAD models or dense reference image sets, which are impractical for spontaneous interaction.

HIPPo overview comparing existing 6D pose and SLAM methods with the proposed approach.

The Problem

Current "zero-shot" methods often suffer from slow reconstruction times or produce incomplete models (e.g., single-view point clouds) that are insufficient for complex tasks like collision-aware grasping. Furthermore, generative models that create 3D content from images are typically scale-agnostic, whereas robotics requires absolute physical dimensions in meters or centimeters.

Methodology

HIPPo (Harnessing Image-to-3D Priors) addresses these issues through a three-stage pipeline designed for speed and accuracy:

The HIPPo pipeline: HIPPo Dreamer for initial mesh, Pose Estimation Network for tracking, and Mesh Update Module for refinement.

1. HIPPo Dreamer

Starting from a single RGB-D "first glance," this module uses a multiview diffusion model (Wonder3D) to hallucinate the unobserved sides of the object. These views are converted into a 3D point cloud using a reconstruction foundation model (MASt3R). A critical innovation is Scale Recovery: by aligning the generated mesh with the depth data from the initial frame, HIPPo ensures the model matches real-world physical dimensions.

2. Pose Estimation and Tracking

The system employs a refinement network (based on FoundationPose) to compare the generated 3D mesh against incoming camera frames. This allows for real-time 6D pose tracking at approximately 15 frames per second (FPS).

3. Mesh Update Module

As the robot moves and observes new angles, the initial "dreamed" mesh is updated. Using a measurement-guided scheme, the system identifies keyframes and replaces the diffusion-generated "priors" with reliable real-world measurements via Poisson surface reconstruction. This ensures the model evolves from a plausible guess to a high-fidelity representation.

Key Insights

  • Priors as a "Warm Start": Diffusion models allow the robot to act immediately with a "complete enough" model rather than waiting for a full SLAM-based reconstruction.
  • Bridging the Scale Gap: Using the first-frame depth as an anchor transforms scale-agnostic generative outputs into actionable robotic models.
  • Strategic Refinement: The system prioritizes real measurements over hallucinations as they become available, tracked via a "viewpoint sphere" to manage observed vs. unobserved surfaces.

Results and Evaluation

HIPPo was evaluated on standard benchmarks like YCB-Video and LM-O, as well as a custom robotic dataset using a Franka Emika arm.

  • Performance: On YCB-Video with a single reference image, HIPPo achieved an ADD-S score of 97.00, significantly outperforming previous SOTA methods like FoundationPose (79.71) and SAM-6D (40.62).
  • Efficiency: The initial mesh is generated in approximately 8 seconds, and tracking runs at 15 FPS, making it suitable for immediate robotic applications like grasping and collision avoidance.

Conclusion

HIPPo demonstrates that generative AI priors can be effectively harnessed for precise robotic tasks. By combining the creative power of diffusion models with rigorous real-time measurement refinement, HIPPo enables robots to interact with the unknown with unprecedented speed and accuracy. Future work will focus on improving robustness to severe occlusion and optimizing the computational overhead of the mesh update process.

Summary updated

BibTeX
@article{liu2025hippo,
  author = {Yibo Liu and Zhaodong Jiang and Binbin Xu and Guile Wu and Yuan Ren and Tongtong Cao and Bingbing Liu and Rui Heng Yang and Amir Rasouli and Jinjun Shan},
  title = {{HIPPo: Harnessing Image-to-3D Priors for Model-Free Zero-Shot 6D Pose Estimation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  doi = {10.1109/lra.2025.3585384},
}
Figure