Object Goal Navigation using Goal-Oriented Semantic Exploration

Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan Salakhutdinov

Introduction

Imagine being dropped into a house you've never seen before and told to "find the dining table." As a human, you don't wander randomly; you use semantic priors. You know that dining tables are usually in dining rooms, which are often near kitchens. Traditional autonomous navigation often relies on "end-to-end" reinforcement learning, where a robot tries to learn everything—from pixels to motor commands—all at once. These methods usually fail at Object Goal Navigation because they struggle with long-term planning and tend to memorize specific training layouts rather than learning general rules about how houses are organized.

To bridge this gap, researchers developed Goal-Oriented Semantic Exploration (SemExp), a modular navigation system that masters Object Goal Navigation by building explicit semantic maps and learning where objects are likely to be found. By combining pre-trained computer vision with a high-level policy that understands spatial relationships, SemExp won the 2020 Habitat ObjectNav Challenge and successfully transferred from simulation to real-world robots.

The SemExp Architecture

The authors propose breaking the complex task of navigation into three distinct, manageable modules:

SemExp System Architecture

1. Semantic Mapping

Instead of just seeing a meaningless cloud of points, the robot uses an RGB-D camera and a pre-trained model (Mask R-CNN) to identify objects in its environment.

Semantic Mapping Pipeline

It then projects these 3D "semantic labels" onto a 2D top-down map. This map continuously tracks obstacles, explored areas, and the locations of specific object categories (like chairs, beds, or toilets) as the robot moves.

2. Goal-Oriented Semantic Policy

This module acts as the "brain" of the operation. It looks at the current semantic map and decides on a long-term goal. If the target object (e.g., a "bed") hasn't been seen yet, the policy uses its learned semantic priors to pick an area that is likely to contain a bedroom. It is trained using Reinforcement Learning (Proximal Policy Optimization, or PPO) but operates at a coarse time scale (making a decision every 25 steps) to simplify the learning process and enable long-horizon planning.

3. Deterministic Local Policy

Once a long-term goal is set by the semantic policy, a standard analytical planner (using the Fast Marching Method) calculates the most efficient path to get there. This deterministic policy handles the low-level, step-by-step obstacle avoidance.

Key Insights

  • Modularity is the Secret Sauce: By separating "where am I?" (mapping) from "where should I go?" (high-level policy) and "how do I get there?" (local planning), the system becomes much more robust and easier to train than monolithic end-to-end models.
  • Semantic Priors are Essential: The model doesn't just look for a "chair"; it learns that chairs are often near tables. This allows for "informed exploration"—the robot searches the most promising areas first, significantly reducing the time required to find a target.
  • Seamless Sim-to-Real Transfer: Because the high-level policy operates on a top-down semantic map (which looks structurally similar whether generated from a simulation or a real camera), the model transfers to real-world robots with surprising ease, bypassing the "visual domain gap" that plagues many other methods.

Experimental Results

The researchers evaluated SemExp using the Habitat simulator across two major 3D environment datasets: Gibson (visually realistic reconstructions of real-world homes) and Matterport3D (larger, more complex indoor environments).

  • Challenge Winner: SemExp was the winning entry of the CVPR 2020 Habitat ObjectNav Challenge, outperforming all other submissions.
  • Superior Efficiency: On the Gibson dataset, SemExp achieved a success rate of 54.4%, significantly higher than the "Active Neural SLAM" baseline (44.6%) and drastically outperforming standard end-to-end RL methods (8.2%).
  • Real-World Success: The model was deployed on a LoCoBot mobile robot platform to test its performance in a physical setting. In 20 real-world trials, the system achieved a 65% success rate, proving that the modular design effectively handles noisy real-world sensors and movement dynamics.

Limitations and Future Work

While highly effective, the system has notable failure modes. First, perception errors can derail the process; if the object detection model (Mask R-CNN) fails to see an object or misidentifies it, the semantic map becomes inaccurate, leading the robot astray. Second, in 27% of failed cases, the robot experienced search timeouts, simply failing to find the object within the allotted time limit. This suggests that even better exploration strategies or more efficient mapping techniques could be developed.

Future work could involve extending the system to handle a sequence of object goals (e.g., "find the keys, then the door") by better utilizing the episodic memory built during the initial search phase.

Summary updated

Abstract

This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. End-to-end learning-based navigation methods struggle at this task as they are ineffective at exploration and long-term planning. We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category. Empirical results in visually realistic simulation environments show that the proposed model outperforms a wide range of baselines including end-to-end learning-based methods as well as modular map-based methods and led to the winning entry of the CVPR-2020 Habitat ObjectNav Challenge. Ablation analysis indicates that the proposed model learns semantic priors of the relative arrangement of objects in a scene, and uses them to explore efficiently. Domain-agnostic module design allow us to transfer our model to a mobile robot platform and achieve similar performance for object goal navigation in the real-world.
BibTeX
@inproceedings{chaplot2020object,
  author = {Devendra Singh Chaplot and Dhiraj Gandhi and Abhinav Gupta and Ruslan Salakhutdinov},
  title = {{Object Goal Navigation using Goal-Oriented Semantic Exploration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=VsEm_kvWIk},
}
Figure