Robotic Occlusion Reasoning for Efficient Object Existence Prediction

Mengdi Li, Cornelius Weber, Matthias Kerzel, Jae Hee Lee, Zheni Zeng, Zhiyuan Liu, Stefan Wermter

Introduction

For indoor assistant robots to be genuinely useful, they must efficiently locate objects based on verbal commands. A significant challenge in this domain is object occlusion, where target items are hidden behind other visible entities. Efficiently determining whether a queried object exists in an environment requires sophisticated reasoning about physical space and object dimensions.

Problem Statement: The ROEP Task

The authors introduce the Robotic Object Existence Prediction (ROEP) task. In this scenario, a robot must decide if a queried object is present despite potential occlusions. Reasoning about occlusion is complex; the robot must compare the size of the target object (derived from a verbal cue) with visible objects to determine if the target could be physically hidden.

A robot queried about the existence of a 'marble' in a scene with potential occlusion.

If occlusion is possible, the robot must move to inspect the hidden space; otherwise, it should immediately predict the object's absence to conserve time and energy. While active perception has been studied for navigation, minimizing movement steps while maintaining high accuracy in occlusion reasoning remains a novel challenge.

Methodology

The proposed solution is a recurrent neural network (RNN) architecture comprising five modules: Visual Perception (ResNet18-based), Word Embedding for verbal queries, a temporal Memory unit, an RL-based Action Selection agent, and an Existence Prediction classifier.

Egocentric view sequence showing the robot performing active perception to reveal a hidden object.

Training Strategy

The model employs a hybrid training approach. The Action Selection module is trained via the REINFORCE algorithm with a cost-sensitive reward function that penalizes excessive movement. Other modules use standard supervised learning with binary cross-entropy. To stabilize training, a four-stage curriculum is used, progressing from single objects to complex scenes with occlusions.

Key Results and Insights

The model demonstrates remarkable efficiency and reasoning capabilities:

  • High Efficiency: On the overall test set, the model achieves 97.2% accuracy with an average of only 0.71 movement steps. This matches the accuracy of an "exhaustive exploration" baseline while requiring only 11.8% of the movement steps.
  • Implicit Physical Reasoning: The model learns to relate linguistic concepts to physical sizes. For instance, it correctly predicts "no" immediately if a large object is queried but only small objects are visible, recognizing that a small object cannot hide a large one.
  • Generalization: Tested on a holdout set of 42 unseen object combinations, the model maintained 86.7% accuracy on occluded scenes, indicating it learned general rules of size and occlusion rather than memorizing specific pairs.

Conclusion and Future Work

This research highlights that active perception is essential for handling occlusions, as passive baselines perform significantly worse (~77-83% accuracy). While successful, the model currently exhibits some "lazy" behaviors, such as always circling in one direction. Future work will focus on more complex environments, including more objects, counting tasks, and transitioning from simulation to real-world deployment.

Summary updated

BibTeX
@inproceedings{li2021robotic,
  author = {Mengdi Li and Cornelius Weber and Matthias Kerzel and Jae Hee Lee and Zheni Zeng and Zhiyuan Liu and Stefan Wermter},
  title = {{Robotic Occlusion Reasoning for Efficient Object Existence Prediction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9635947},
}
Figure