RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering

Shun Iwase, Xingyu Liu, Rawal Khirodkar, Rio Yokota, Kris M. Kitani

Abstract

We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the computational cost of CNN, which is especially prominent in multiple-object pose refinement. To overcome this problem, RePOSE leverages image rendering for fast feature extraction using a 3D model with a learnable texture. We call this deep texture rendering, which uses a shallow multilayer perceptron to directly regress a view-invariant image representation of an object. Furthermore, we utilize differentiable Levenberg-Marquardt (LM) optimization to refine a pose fast and accurately by minimizing the distance between the input and rendered image representations without the need of zooming in. These image representations are trained such that differentiable LM optimization converges within few iterations. Consequently, RePOSE runs at 92 FPS and achieves state-of-the-art accuracy of 51.6% on the Occlusion LineMOD dataset - a 4.1% absolute improvement over the prior art, and comparable result on the YCB-Video dataset with a much faster runtime.

Introduction

Determining the 6D pose—the 3D rotation and 3D translation—of objects is a fundamental task in computer vision, critical for applications in robotics and augmented reality. While initial pose detectors provide a coarse estimate, achieving the precision required for tasks like robotic grasping often necessitates an iterative refinement step.

Current state-of-the-art refinement methods, such as DeepIM or CosyPose, rely on heavy Convolutional Neural Networks (CNNs). These methods typically crop the input image around the object and compare it with a rendered RGB image of the 3D model to predict a pose update. However, this "zoom-and-regress" approach is computationally expensive, leading to slow runtimes that scale poorly with the number of objects in a scene. Furthermore, traditional photometric alignment methods often struggle with varying lighting conditions or textureless objects.

Methodology: Deep Texture Rendering and LM Optimization

RePOSE addresses these efficiency and robustness challenges by decoupling feature extraction from the iterative loop and replacing direct pose regression with geometric optimization.

The RePOSE pipeline: Iterative pose refinement using deep texture rendering and differentiable LM optimization.

The architecture consists of three core pillars:

  1. Static Feature Extraction: A U-Net processes the input RGB image once to generate a dense feature map ($F_{inp}$). Unlike previous methods that re-run CNNs on every iteration, this map remains fixed, significantly reducing the per-iteration cost.
  2. Deep Texture Renderer: Instead of standard RGB rendering, RePOSE uses "learnable textures" (high-dimensional vectors) assigned to the vertices of the object's 3D mesh. A shallow MLP and a custom renderer project these into a 2D feature map ($F_{rend}$) in under 1ms. These features are designed to be view-invariant and robust to environmental changes.
  3. Differentiable Levenberg-Marquardt (LM) Optimization: The system minimizes the distance between $F_{inp}$ and $F_{rend}$ using the LM algorithm. Because the entire pipeline is differentiable, the deep textures and the U-Net can be trained end-to-end to ensure the optimizer converges rapidly.

Experimental Evaluation

The authors evaluated RePOSE on three benchmark datasets: LineMOD (standard objects in clutter), Occlusion LineMOD (highly occluded objects), and YCB-Video (household objects for tracking).

Performance and Speed

RePOSE achieves a remarkable throughput of 92 FPS for a single object. Crucially, its efficiency allows it to maintain over 60 FPS even with 10 objects in a scene, whereas competitors like DeepIM drop to approximately 6 FPS for just 5 objects.

Accuracy

On the challenging Occlusion LineMOD dataset, RePOSE set a new state-of-the-art with 51.6% accuracy (ADD-S metric), representing a 4.1% absolute improvement over previous methods. This performance demonstrates that geometric optimization on learned features is more robust than direct regression, especially under heavy occlusion.

Qualitative results on the Occlusion LineMOD dataset showing successful pose refinement across various objects.

Discussion and Conclusion

RePOSE represents a significant shift in pose refinement by prioritizing efficiency through decoupling and robustness through geometry. By using deep textures, the model becomes more data-efficient, requiring significantly fewer training images than regression-based counterparts to reach similar performance levels.

However, some limitations remain. RePOSE is a refiner and thus depends on a reasonably accurate initial pose from a detector. If the initial estimate is too far from the ground truth, the LM optimizer may converge to a local minimum. Additionally, the method requires a pre-existing 3D mesh of the target object. Future work may explore multi-scale feature representations to improve sensitivity for very small objects, where sub-pixel accuracy is paramount.

Summary updated

Abstract

We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the computational cost of CNN, which is especially prominent in multiple-object pose refinement. To overcome this problem, RePOSE leverages image rendering for fast feature extraction using a 3D model with a learnable texture. We call this deep texture rendering, which uses a shallow multilayer perceptron to directly regress a view-invariant image representation of an object. Furthermore, we utilize differentiable Levenberg-Marquardt (LM) optimization to refine a pose fast and accurately by minimizing the distance between the input and rendered image representations without the need of zooming in. These image representations are trained such that differentiable LM optimization converges within few iterations. Consequently, RePOSE runs at 92 FPS and achieves state-of-the-art accuracy of 51.6% on the Occlusion LineMOD dataset - a 4.1% absolute improvement over the prior art, and comparable result on the YCB-Video dataset with a much faster runtime. The code is available at https://github.com/sh8/repose.
BibTeX
@inproceedings{iwase2021repose,
  author = {Shun Iwase and Xingyu Liu and Rawal Khirodkar and Rio Yokota and Kris M. Kitani},
  title = {{RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=1IGnkWL9wX},
}
Figure