Object Goal Navigation Based on Semantics and RGB Ego View

Snehasis Banerjee, Brojeshwar Bhowmick, Ruddra Dev RoychoudhuryarXiv 2022 · DOI · arXiv

Introduction

Navigating an unmapped indoor environment to locate a specific object—such as a "cup" or an "orange"—is a task humans perform intuitively but remains a significant challenge for robotics. Traditional approaches often rely on expensive LiDAR sensors for geometric mapping or require extensive training data for end-to-end reinforcement learning, which frequently fails to generalize to novel environments. Researchers at TCS Research have proposed a novel architecture that leverages semantic "common sense" and standard RGB cameras to bridge this gap, enabling robots to find targets efficiently in unknown spaces.

System Architecture and Methods

The proposed system centers on the GeoSem Map, a relational framework that integrates geometric data with semantic knowledge. The architecture consists of several interconnected modules designed to process visual information and make high-level navigation decisions.

The proposed navigation architecture, highlighting the flow from RGB perception to the decision module and action space.

Perception and Knowledge Representation

The perception pipeline utilizes a "Frame Filter" to select keyframes from the video stream. These frames are processed by YOLO v3 for fast object detection, with a fallback to Mask-RCNN for more complex scenes. Instead of treating the environment as a collection of pixels, the robot interprets it through semantic relationships stored in a knowledge base using OWL and RDF technologies. By training on the Visual Genome dataset, the robot understands probabilistic co-locations, such as a "cup" likely being found on a "table."

Decision Logic and the Landmark Score

To guide navigation, the system calculates a "Landmark Score," a heuristic that assesses the probability of a target's presence based on current visual cues and exploration history. If the score is low, the robot may backtrack to a previous high-probability zone. The decision module prioritizes moving through doors or toward "relational objects" (e.g., moving toward a table when searching for a cup) to narrow the search space.

Experimental Setup

The system was evaluated in both simulated and real-world environments.

The Webots simulation environment used for gamified evaluation, featuring the robot's ego camera view.

The researchers created a custom indoor environment in Webots for a "gamified" comparison between the robot and human participants. Real-world testing was conducted using a Double2 wheeled robot in a standard office setting, where it successfully navigated to find targets like a "cup" hidden behind a "chair" by identifying intermediate cues like "bottles" and "tables."

Key Results and Insights

The "gamified" evaluation revealed that the robot significantly outperformed human users in a search-and-find task. While human participants (IT professionals) took an average of 632 seconds to find an "orange" in the simulation, the robot completed the same task in just 117 seconds.

This superior performance is attributed to:

  • Semantic Reasoning: The robot uses a knowledge graph to follow a "semantic trail" (e.g., chair -> table -> cup) rather than wandering aimlessly.
  • Occlusion Handling: The system explicitly models occlusionBy relations, allowing the robot to infer that a target might be hidden and plan its path to gain a better vantage point.
  • Systematic Exploration: Unlike humans, who often struggled with spatial orientation in a 2D interface, the robot maintained a consistent and efficient exploration strategy.

Limitations and Future Work

Despite its success, the current model assumes a static environment and does not account for dynamic obstacles like moving people. Additionally, the reliance on pure RGB data can lead to challenges in precise obstacle avoidance compared to depth-sensing systems. Future work will focus on integrating "Stream Reasoning" for faster real-time decision-making and incorporating depth estimation to enhance navigation in complex, dynamic real-world settings.

Summary updated

Abstract

This paper presents an architecture and methodology to empower a service robot to navigate an indoor environment with semantic decision making, given RGB ego view. This method leverages the knowledge of robot's actuation capability and that of scenes, objects and their relations -- represented in a semantic form. The robot navigates based on GeoSem map - a relational combination of geometric and semantic map. The goal given to the robot is to find an object in a unknown environment with no navigational map and only egocentric RGB camera perception. The approach is tested both on a simulation environment and real life indoor settings. The presented approach was found to outperform human users in gamified evaluations with respect to average completion time.
BibTeX
@misc{banerjee2022object,
  author = {Snehasis Banerjee and Brojeshwar Bhowmick and Ruddra Dev Roychoudhury},
  title = {{Object Goal Navigation Based on Semantics and RGB Ego View}},
  year = {2022},
  eprint = {2210.11543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.11543},
}
Figure