Active Object Search

Jie Wu, Tianshui Chen, Lishan Huang, Hefeng Wu, Guanbin Li, Ling Tian, Liang Lin

Abstract

Active Object Search (AOS) is a novel task requiring an intelligent agent to navigate a 3D environment to locate a target object with minimal steps. We present a modular reinforcement learning framework that integrates a 3D object detector, a specialized state controller, and a cost-sensitive reward function. Our approach achieves efficient, human-like search behavior by balancing detection accuracy with movement economy.

Introduction

Traditional object detection systems are typically passive, assuming the target is already centered in a static image. In real-world robotics, however, objects are often occluded, distant, or entirely out of view. An "active" agent must reason about its environment and plan movements—such as rotating or moving closer—to bring the target into focus.

Comparison of Agent and Third-Person perspectives highlighting search challenges: (1) far from camera, (2) blocked by obstacle, (3) out of view.

The core challenge in AOS is optimizing the trade-off between search accuracy and efficiency. An agent must find the object correctly without unnecessary exploration.

Methodology

Our reinforcement learning framework consists of three primary components:

  1. 3D Object Detector: Uses Frustum PointNets on RGB-D data to predict 3D bounding boxes.
  2. State Controller: Monitors detection confidence (IoU) to emit a "stop" signal only when certainty is high, significantly reducing termination errors.
  3. Cross-modal Action Planner: A ResNet-18 and LSTM-based "brain" that processes visual scenes and target queries to decide the next action.

To encourage efficiency, we employ a Cost-Sensitive Active Search Reward (CSASR). The agent is rewarded for successful detection but penalized for every action taken, forcing it to find the most direct path.

The AOS Benchmark

We introduced the AOS benchmark using the AI2-THOR platform, featuring 5,845 samples across 30 indoor scenes. The dataset includes increased randomness in object placement and room layouts, categorized into three difficulty tiers:

  • Easy: Target is initially in view.
  • Moderate: Target is nearby but out of view.
  • Hard: Target is distant, requiring multiple complex actions.

Results and Discussion

The CSASR method significantly outperforms passive and standard active baselines. While a naive static detector fails in Moderate and Hard scenarios, CSASR maintains a 50.53% recall in the most difficult settings.

A visualization of the agent's sequential search trajectory (steps 1-5) to locate a target object.

Key insights include:

  • Explicit Termination: A dedicated state controller reduced termination errors from 46% to 10%.
  • Cost-Sensitivity: Penalizing movement mimics human-like efficiency, avoiding redundant actions.
  • 3D Spatial Reasoning: 3D detection is vital for understanding depth and scale, enabling better navigation than 2D-only approaches.

Conclusion

AOS represents a significant step toward practical robotic vision. Future work will focus on sim-to-real transfer, multi-room environments, and integrating search with complex downstream tasks like object manipulation.

Summary updated

Abstract

In this work, we investigate an Active Object Search (AOS) task that is not explicitly addressed in the literature. It aims to actively perform as few action steps as possible to search and locate the target object in a 3D indoor scene. Different from classic object detection that passively receives visual information, this task encourages an intelligent agent to perform active search via reasonable action planning; thus it can better recall the target objects, especially for the challenging situations that the target is far from the agent, blocked by an obstacle and out of view. To handle this AOS task, we formulate a reinforcement learning framework that consists of a 3D object detector, a state controller and a cross-modal action planner to work cooperatively to find out the target object with minimal action steps. During training, we design a novel cost-sensitive active search reward that penalizes inaccurate object search and redundant action steps. To evaluate this novel task, we construct an Active Object Search (AOS) benchmark that contains 5,845 samples from 30 diverse indoor scenes. We conduct extensive qualitative and quantitative evaluations on this benchmark to demonstrate the effectiveness of the proposed approach and analyze the key factors that contribute more to address this task.
BibTeX
@inproceedings{wu2020active,
  author = {Jie Wu and Tianshui Chen and Lishan Huang and Hefeng Wu and Guanbin Li and Ling Tian and Liang Lin},
  title = {{Active Object Search}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413571},
}
Figure