Occlusion-Robust Object Pose Estimation with Holistic Representation

Bo Chen, Tat-Jun Chin, Marius Klimavicius

TL;DR

This paper introduces ROPE, a novel framework for robust 6DOF object pose estimation from single RGB images, specifically tackling the challenge of occlusions. By combining a new "occlude-and-blackout" data augmentation technique with a multi-precision supervision strategy for landmark prediction, ROPE achieves state-of-the-art performance without requiring post-processing or refinement steps.

What Problem Does This Paper Address?

Estimating the 3D pose (position and orientation) of an object from a 2D image is a fundamental problem in computer vision with applications ranging from robotics to augmented reality. A major hurdle in this task is occlusion, where parts of the object are hidden from view by other objects or scene elements. Existing state-of-the-art methods often employ a two-stage approach: first, predicting 2D landmarks on the object, and second, using these correspondences to solve for the 6DOF pose. However, these methods can struggle with novel occlusions, leading to inconsistent landmark predictions and ultimately inaccurate pose estimates. Furthermore, current approaches often rely on separate refinement steps to improve accuracy, adding computational overhead. This paper aims to develop a more inherently robust base pose estimator that can handle occlusions effectively without needing post-processing.

Methods

The ROPE framework builds upon the Mask R-CNN architecture, replacing its mask head with three specialized keypoint heads. The core innovations are:

  • Occlude-and-Blackout Batch Augmentation (OBA): This novel technique extends standard data augmentations by randomly occluding and blacking out parts of the object within training batches. This forces the network to learn features that are invariant to occlusions and background clutter, promoting robust landmark prediction.
  • Multi-Precision Supervision (MPS): Instead of using a single groundtruth heatmap for each landmark, MPS employs three keypoint heads that predict heatmaps with varying variances (low, medium, and high precision). This encourages the network to learn a more holistic representation of the object's shape and structure, improving landmark coherence and the ability to infer occluded parts. The high-precision heatmaps are used for final pose estimation, while the medium-precision ones are used for a simple landmark filtering step.
  • Landmark Filtering: A lightweight filtering mechanism uses the medium-precision landmark predictions to verify the high-precision ones. Only landmarks with consistent predictions across different precision levels are passed to the PnP solver, reducing the need for complex post-refinement.

Datasets

The paper evaluates ROPE on three widely used datasets:

  • LINEMOD: A standard dataset for object pose estimation.
  • Occluded-LINEMOD: An extension of LINEMOD specifically designed to test robustness to occlusions.
  • YCB-Video: A dataset containing real-world video sequences of objects.

The authors follow standard protocols for training and testing on these datasets, including the use of rendered synthetic images for LINEMOD and Occluded-LINEMOD.

Key Insights and Ideas

  • Holistic Representation is Key for Occlusion Robustness: The paper strongly suggests that learning a "holistic" understanding of the object, where different parts of the network learn to relate to each other, is crucial for handling occlusions. This goes beyond just predicting individual landmarks accurately.
  • Data Augmentation Can Be More Than Just Noise: The OBA technique demonstrates that strategically introducing occlusions during training can directly lead to more robust feature learning, rather than just relying on general robustness from diverse data.
  • Multi-Precision Supervision Boosts Coherence: The MPS approach highlights that training with different levels of detail (heatmap variances) can implicitly teach the network about the object's structure, leading to more consistent and spatially accurate landmark predictions, especially when parts are occluded.
  • Eliminating Refinement is Achievable: The results show that a well-designed base estimator can achieve competitive or superior performance to methods that rely on post-processing refinement, offering a significant advantage in terms of speed and efficiency.

Key Results

  • State-of-the-Art Performance: ROPE achieves superior performance on the Occluded-LINEMOD dataset and outperforms all non-refinement methods on the YCB-Video dataset in terms of the ADD(-S) metric.
  • Competitive with Refinement Methods: Even without any post-processing or refinement, ROPE is competitive with state-of-the-art methods that do include refinement.
  • High Data Efficiency: ROPE demonstrates remarkable data efficiency, achieving strong results with significantly fewer training images compared to methods that rely heavily on synthetic data generation. For example, it achieves an average ADD(-S) of 93.22% on LINEMOD using only ~180 training images.
  • Qualitative Improvements: Visual results show that ROPE produces more accurate and structurally consistent landmark predictions, especially in challenging occluded scenarios, compared to baseline models.

Limitations and Future Work

  • Limitations: The paper focuses on the base pose estimator and does not include a refinement step. While competitive, further improvements might be possible by integrating ROPE with existing refinement techniques. The landmark filtering step is a simple verification, and more sophisticated methods could potentially be explored.
  • Future Work: The authors suggest that their pipeline can be easily extended to incorporate one or multiple refinement stages. They also implicitly point towards further exploration of data augmentation strategies and multi-precision learning for robust vision tasks.

Summary updated

Abstract

Practical object pose estimation demands robustness against occlusions to the target object. State-of-the-art (SOTA) object pose estimators take a two-stage approach, where the first stage predicts 2D landmarks using a deep network and the second stage solves for 6DOF pose from 2D-3D correspondences. Albeit widely adopted, such two-stage approaches could suffer from novel occlusions when generalising and weak landmark coherence due to disrupted features. To address these issues, we develop a novel occlude-and-blackout batch augmentation technique to learn occlusion-robust deep features, and a multi-precision supervision architecture to encourage holistic pose representation learning for accurate and coherent landmark predictions. We perform careful ablation tests to verify the impact of our innovations and compare our method to SOTA pose estimators. Without the need of any post-processing or refinement, our method exhibits superior performance on the LINEMOD dataset. On the YCB-Video dataset our method outperforms all non-refinement methods in terms of the ADD(-S) metric. We also demonstrate the high data-efficiency of our method. Our code is available at http://github.com/BoChenYS/ROPE
BibTeX
@inproceedings{chen2022occlusion,
  author = {Bo Chen and Tat-Jun Chin and Marius Klimavicius},
  title = {{Occlusion-Robust Object Pose Estimation with Holistic Representation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=mjWxvzKx6ln},
}
Figure