Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation

Xiaohan Lei, Min Wang, Wengang Zhou, Li Li

Introduction

Instance ImageGoal Navigation (IIN) is a challenging task in embodied AI where an agent must locate a specific object instance within a complex environment, guided only by a goal image. Unlike standard ImageNav, which targets object categories, IIN requires distinguishing the target from visually similar distractors—for example, finding a specific bed with unique pillows among several beds in a house. Existing approaches often struggle with this ambiguity, frequently stopping at the first similar-looking object they encounter.

The IEVE Framework

To address these challenges, the Instance-aware Exploration-Verification-Exploitation (IEVE) framework introduces a dedicated "Verification" phase between exploration and exploitation. This mimics the human behavior of approaching a potential target to confirm its identity before committing to it as the goal.

The IEVE framework in action: the agent explores the environment, verifies a potential distractor at $t_2$, and finally exploits the confirmed target at $t_3$.

The framework consists of five modular components:

  1. Instance Classification: Determines the target's category from the goal image.
  2. Online Mapping: Constructs a real-time 2D semantic map using RGB-D data and instance segmentation (e.g., InternImage).
  3. Switch Policy: The core logic that toggles between Exploration (searching for the category), Verification (approaching a candidate), and Exploitation (navigating to a confirmed instance).
  4. Goal Mapping Policy: Sets long-term coordinates based on the current state.
  5. Local Policy: Executes low-level navigation to reach the chosen coordinates.

Distance-Aware Verification

The "Verification" step utilizes local feature matching (DISK and LightGlue) to compare the agent's current view with the goal image. A critical innovation is the use of an adaptive threshold for confirmation. Because visual matching reliability is distance-dependent, the agent requires more stringent matching criteria when far away. As the agent moves closer, the threshold relaxes, allowing for a robust confirmation process that effectively filters out distractors.

Results and Discussion

Evaluated on the HM3D-SEM dataset within the Habitat simulator, IEVE achieves a state-of-the-art success rate of 70.2%, significantly outperforming the previous best of 56.1%. Notably, the framework demonstrates extreme efficiency compared to end-to-end reinforcement learning methods, which achieve less than 10% success even after billions of training steps.

While IEVE is highly effective, it remains dependent on the accuracy of the underlying segmentation models. Future research will explore incorporating high-level semantic reasoning—such as understanding that beds are typically found in bedrooms—to further optimize the exploration phase and adapt the system for dynamic real-world environments.

Summary updated

Abstract

As a new embodied vision task, Instance ImageGoal Navigation (IIN) aims to navigate to a specified object depicted by a goal image in an unexplored environment. The main challenge of this task lies in identifying the target object from different viewpoints while rejecting similar distractors. Existing ImageGoal Navigation methods usually adopt the simple Exploration-Exploitation framework and ignore the identification of specific instance during navigation. In this work, we propose to imitate the human behaviour of ``getting closer to confirm" when distinguishing objects from a distance. Specifically, we design a new modular navigation framework named Instance-aware Exploration-Verification-Exploitation (IEVE) for instance-level image goal navigation. Our method allows for active switching among the exploration, verification, and exploitation actions, thereby facilitating the agent in making reasonable decisions under different situations. On the challenging HabitatMatterport 3D semantic (HM3D-SEM) dataset, our method surpasses previous state-of-the-art work, with a classical segmentation model (0.684 vs. 0.561 success) or a robust model (0.702 vs. 0.561 success)
BibTeX
@inproceedings{lei2024instance,
  author = {Xiaohan Lei and Min Wang and Wengang Zhou and Li Li and Houqiang Li},
  title = {{Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01545},
}
Figure