Object Goal Navigation using Goal-Oriented Semantic Exploration

Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan Salakhutdinov

Abstract This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. End-to-end learning-based navigation methods struggle at this task as they are ineffective at exploration and long-term planning. We propose a modular system called, 'Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category. Empirical results in visually realistic simulation environments show that the proposed model outperforms a wide range of baselines including end-to-end learning-based methods as well as modular map-based methods and led to the winning entry of the CVPR-2020 Habitat ObjectNav Challenge. Ablation analysis indicates that the proposed model learns semantic priors of the relative arrangement of objects in a scene, and uses them to explore efficiently. Domain-agnostic module design allow us to transfer our model to a mobile robot platform and achieve similar performance for object goal navigation in the real-world.

Introduction

Navigating a robot through a brand-new house to find a specific object—like a "dining table" or a "toilet"—is a deceptively complex task. It requires more than just avoiding walls; it requires "common sense" about how human environments are organized. You wouldn't look for a bed in a kitchen, and a robot shouldn't either. Researchers from Carnegie Mellon and Facebook AI Research present SemExp (Goal-Oriented Semantic Exploration), a modular system that won the CVPR 2020 Habitat ObjectNav Challenge by teaching robots to use semantic "priors" to explore efficiently.

The Modular Architecture

The core of the SemExp system is its modularity. Unlike traditional "end-to-end" reinforcement learning (RL) models that attempt to learn pixels-to-actions directly, SemExp decomposes the problem into three distinct components: mapping, high-level planning, and low-level execution.

The modular architecture of the SemExp system, showing the flow from sensor input to action.
  1. Semantic Mapping: Builds a top-down episodic map with channels for different object categories.
  2. Goal-Oriented Semantic Policy: Decides on a long-term goal coordinate based on the current map and learned semantic priors.
  3. Deterministic Local Policy: Uses a classical path planner to calculate the shortest path to the long-term goal while avoiding obstacles.

Semantic Mapping

As the robot moves, it uses RGB-D images and a pretrained Mask R-CNN to detect objects. These 2D detections are projected into 3D space using differentiable geometric operations to build a "top-down" semantic map. Unlike a standard occupancy map that just shows obstacles, this map has specific "channels" for different object categories (e.g., chairs, beds, etc.). This allows the agent to maintain an explicit memory of what it has seen and where.

The semantic mapping pipeline, converting RGB-D observations into a multi-channel top-down map via point clouds and voxels.

Goal-Oriented Semantic Policy

This is the "brain" of the system. It looks at the current semantic map and decides on a long-term goal. If the target object hasn't been seen yet, the policy uses learned semantic priors to pick a spot where the object is likely to be. For example, if the robot sees a fridge, it learns that it is in a kitchen and that a toaster or microwave is likely nearby. This policy is trained via Reinforcement Learning (PPO) but operates at a coarse time scale (every 25 steps), which simplifies the learning process and focuses on high-level strategy rather than low-level motor control.

Key Results

SemExp demonstrated significant improvements over existing methods in both simulation and real-world testing:

  • Challenge Winner: SemExp won the CVPR 2020 Habitat ObjectNav Challenge, achieving a 25.3% success rate compared to the 18.8% of the runner-up.
  • Efficiency in Simulation: On the Gibson dataset, SemExp achieved a 54.4% success rate, vastly outperforming standard end-to-end RL, which managed only 8.2%.
  • Real-World Transfer: When deployed on a physical Locobot, the system maintained a 65% success rate across 20 trials. Because the high-level policy operates on top-down maps rather than raw pixels, the model transfers to real-world hardware with minimal "reality gap" friction.

Conclusion

By separating high-level semantic reasoning from low-level navigation, SemExp provides a robust framework for ObjectNav. The system's ability to learn and apply semantic priors—understanding that certain objects belong in certain rooms—allows for "informed exploration" that mimics human-like common sense. Future work will look toward handling dynamic environments and more complex, sequential navigation tasks.

Summary updated

Abstract

This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. End-to-end learning-based navigation methods struggle at this task as they are ineffective at exploration and long-term planning. We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category. Empirical results in visually realistic simulation environments show that the proposed model outperforms a wide range of baselines including end-to-end learning-based methods as well as modular map-based methods and led to the winning entry of the CVPR-2020 Habitat ObjectNav Challenge. Ablation analysis indicates that the proposed model learns semantic priors of the relative arrangement of objects in a scene, and uses them to explore efficiently. Domain-agnostic module design allow us to transfer our model to a mobile robot platform and achieve similar performance for object goal navigation in the real-world.
BibTeX
@inproceedings{chaplot2020object,
  author = {Devendra Singh Chaplot and Dhiraj Gandhi and Abhinav Gupta and Ruslan Salakhutdinov},
  title = {{Object Goal Navigation using Goal-Oriented Semantic Exploration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=VsEm_kvWIk},
}
Figure