Abstract
Passive visual systems typically fail to recognize objects in the amodal setting where they are heavily occluded. In contrast, humans and other embodied agents have the ability to move in the environment and actively control the viewing angle to better understand object shapes and semantics. In this work, we introduce the task of Embodied Amodal Recognition (EAR): an agent is instantiated in a 3D environment close to an occluded target object, and is free to move in the environment to perform object classification, amodal object localization, and amodal object segmentation. To address this problem, we develop a new model called Embodied Mask R-CNN for agents to learn to move strategically to improve their visual recognition abilities. We conduct experiments using a simulator for indoor environments. Experimental results show that: 1) agents with embodiment (movement) achieve better visual recognition performance than passive ones and 2) in order to improve visual recognition abilities, agents can learn strategic paths that are different from shortest paths.
Introduction
Standard computer vision systems are typically "passive," attempting to recognize objects from a single, static viewpoint. This becomes exceptionally challenging in cases of heavy occlusion, a problem known as amodal perception. While deep learning has made strides in hallucinating hidden parts of objects, a single 2D perspective is often fundamentally insufficient to resolve complex occlusions.
In contrast, biological agents do not rely solely on static observation. Humans instinctively move—leaning, walking, or rotating—to gain a better vantage point when an object is obscured. This paper introduces Embodied Amodal Recognition (EAR), bridging the gap between computer vision and robotics by enabling an AI agent to move strategically to improve its own visual recognition.
Methods: Embodied Mask R-CNN
To tackle the EAR task, the authors propose Embodied Mask R-CNN, a framework that integrates perception and action. The model consists of two primary components:
1. Amodal Recognition Module (Temporal Mask R-CNN)
This module processes a sequence of images captured as the agent moves. It utilizes a ResNet-50 backbone for feature extraction. To effectively aggregate information over time, the model employs a Convolutional Gated Recurrent Unit (Conv-GRU). This allows the system to maintain a latent representation of the target object, updating it with new visual evidence from multiple viewpoints rather than relying on simple feature averaging.
2. Policy Network
The "moving" component is governed by a policy network trained via Reinforcement Learning (REINFORCE). The agent can choose from six discrete actions: move forward, backward, left, right, or rotate. The policy is rewarded based on the improvement of the final amodal prediction (classification, bounding box, and mask) relative to the ground truth.
Staged Training Strategy
To avoid the "chicken and egg" problem where the policy needs good perception to learn and vice versa, the authors use a three-stage approach:
- Perception Pre-training: The recognition module is trained on data from shortest-path trajectories.
- Policy Training: The policy is trained while the perception module remains fixed.
- Joint Fine-tuning: Both modules are optimized together, allowing the perception model to adapt to the specific strategic paths discovered by the policy.
The EAR Dataset
The researchers developed a large-scale dataset using a 3D indoor simulator to evaluate the EAR task.
- Environment: 550 virtual houses with diverse layouts.
- Objects: 1,331 unique instances across 8 categories (e.g., bed, chair, sofa, fridge).
- Amodal Ground Truth: Unlike standard datasets that clip masks at image borders, EAR provides the full shape of objects, even parts hidden behind walls or outside the camera frame.
- Difficulty Levels: Samples are split into "easy" (visibility > 50%) and "hard" (visibility < 50%) to test the limits of passive vs. active perception.
Key Insights and Results
The experiments compared the proposed "ActivePath" agent against passive and heuristic baselines (Random and Shortest Path).
- Embodiment Outperforms Hallucination: The ActivePath agent achieved a classification accuracy of 93.7%, surpassing the passive baseline of 92.9%. The advantage was most significant in "hard" cases where occlusion was severe.
- Localization & Segmentation: In the "hard" category, the agent improved the Amodal Box IoU (Intersection over Union) from 76.5% to 78.2% and the Amodal Mask IoU from 63.9% to 65.6%.
- Strategic vs. Shortest Paths: Interestingly, the optimal path for recognition is rarely the shortest path to the object. Agents often learned to move backward or sideways to increase their field of view, allowing them to see both the occluding object and the target's boundaries more clearly.
- Temporal Aggregation: The Conv-GRU was critical; simply seeing more frames without a sophisticated way to "warp" and remember features was less effective.
Conclusion and Future Work
This work demonstrates that embodiment—the ability to move—is a powerful tool for visual recognition. By learning to move strategically, agents can resolve ambiguities that are impossible to settle from a single viewpoint.
Future research directions include scaling to more object categories, enabling the agent to handle multiple targets simultaneously, and exploring more complex 3D movements like crouching. Transitioning these learned strategies from simulation to physical robotic platforms remains a key challenge for the field.
Summary updated