VeREFINE: Integrating Object Pose Verification With Physics-Guided Iterative Refinement

Dominik Bauer, Timothy Patten, Markus VinczeIEEE RA-L 2020 · DOI · OpenReview

Abstract

Accurate 6D object pose estimation is critical for robotic manipulation in cluttered environments. However, initial estimates often suffer from inaccuracies due to sensor noise or model discrepancies, and standard refinement techniques frequently ignore physical constraints. We present VeREFINE, a framework that integrates pose verification, iterative refinement, and physics simulation into a unified optimization loop. By utilizing a multi-armed bandit approach for resource allocation, VeREFINE achieves state-of-the-art accuracy and significantly improves grasping success rates in real-world scenarios.

Introduction

For a robot to successfully interact with its environment, it must precisely determine the 6D pose of objects. While deep learning has advanced initial pose estimation, these predictions often require refinement to correct for sensor noise and lighting variations. Traditional pipelines typically employ verification and refinement steps, such as Iterative Closest Point (ICP). However, these methods often produce physically implausible results, such as objects intersecting with surfaces or floating in mid-air.

The VeREFINE framework applied to a Toyota HSR robot for real-world object manipulation.

Physics-based refinement has been proposed to address these issues, but it is often computationally expensive and prone to divergence—where small initial errors lead to unstable simulations. VeREFINE addresses these challenges by interleaving physics simulation with iterative refinement and intelligently managing the computational budget across multiple pose hypotheses.

Methodology

VeREFINE consists of three integrated components designed to ensure physical plausibility and computational efficiency:

  1. Physics-guided Iterative Refinement (PIR): Unlike methods that apply simulation as a post-processing step, PIR interleaves simulation with refinement. A key innovation is the selective use of orientation data from the physics engine. By only updating the object's rotation based on the simulation, the system avoids the translational "jitter" and divergence common in physics-based methods.
  2. Supervised Iterative Refinement (SIR): To guarantee that each refinement step improves the pose estimate, SIR employs a GPU-accelerated verification score ($\bar{f}$). This score compares rendered depth and normals against actual camera observations. Refined poses are only accepted if they yield a higher verification score than their predecessors.
  3. Regret-minimizing Iterative Refinement (RIR): To handle cluttered scenes with numerous hypotheses, RIR treats pose refinement as a Multi-armed Bandit (MAB) problem. Using the Upper Confidence Bound (UCB) algorithm, the system prioritizes the most promising candidates, allowing for an "any-time" response within a fixed time budget.
The three core components of VeREFINE: Physics-guided (PIR), Supervised (SIR), and Regret-minimizing (RIR) Iterative Refinement.

For complex scenes, the framework extends to VeREFINE depth ($VF_d$), which accounts for occlusions and support relationships by processing objects in a structured front-to-back and bottom-up order.

Experimental Results

The framework was rigorously evaluated on standard benchmarks including LINEMOD, Extended APC (xAPC), and YCB-VIDEO, as well as in real-world grasping tasks using a Toyota HSR robot.

Performance and Accuracy

On the LINEMOD dataset, VeREFINE demonstrated a 7.1% improvement in Average Recall (AR) compared to standard refinement techniques. In real-world manipulation tests, the system achieved an 86% grasp success rate, a significant increase over the 62% success rate of standard deep-learning-based refinement. It proved particularly robust for challenging items like foam bricks and bananas.

Computational Efficiency

By implementing verification scores on the GPU, VeREFINE reduces evaluation time to 1-2ms per pose. Consequently, the system is approximately 5x faster than previous state-of-the-art physics-based methods like PHYSIM-MCTS, reducing total processing time per frame from ~30 seconds to under 6 seconds.

Conclusion and Future Work

VeREFINE represents a significant step toward physically-aware robot vision. By combining physics-guided refinement with efficient verification and bandit-based resource management, it provides a robust solution for real-world manipulation. Future research will focus on extending the system to handle unknown objects through shape estimation and adapting to non-planar support structures in complex environments.

Summary updated

Abstract

Accurate and robust object pose estimation for robotics applications requires verification and refinement steps. In this work, we propose to integrate hypotheses verification with object pose refinement guided by physics simulation. This allows the physical plausibility of individual object pose estimates and the stability of the estimated scene to be considered in a unified optimization. The proposed method is able to adapt to scenes of multiple objects and efficiently focuses on refining the most promising object poses in multi-hypotheses scenarios. We call this integrated approach VeREFINE and evaluate it on three datasets with varying scene complexity. The generality of the approach is shown by using three state-of-the-art pose estimators and three baseline refiners. Results show improvements over all baselines and on all datasets. Furthermore, our approach is applied in real-world grasping experiments and outperforms competing methods in terms of grasp success rate. Code is publicly available at github.com/dornik/verefine.
BibTeX
@article{bauer2020verefine,
  author = {Dominik Bauer and Timothy Patten and Markus Vincze},
  title = {{VeREFINE: Integrating Object Pose Verification With Physics-Guided Iterative Refinement}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  doi = {10.1109/lra.2020.2996059},
}
Figure