Learning Navigational Visual Representations with Semantic Map Supervision

Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernoncourt, Trung Bui, Stephen Gould, Hao Tan

Introduction

In the rapidly evolving field of Embodied AI, household robots are increasingly required to navigate complex, unseen indoor environments. While current visual encoders pre-trained on massive datasets like ImageNet or CLIP excel at object recognition, they often lack "spatial common sense." These models are fundamentally spatial-blind; they do not inherently understand the connectivity of rooms, the traversability of spaces, or how their perspective shifts during movement.

To address this, researchers from Adobe and The Australian National University have introduced Ego2-Map. This novel pre-training framework teaches robots to bridge the gap between their first-person (egocentric) observations and the underlying structural layout of their environment by aligning visual features with top-down semantic maps.

Methodology: The Ego2-Map Framework

The core of Ego2-Map is a contrastive learning objective designed to instill navigational awareness into a Vision Transformer (ViT) backbone. Unlike traditional pre-training that relies on static image labels, Ego2-Map utilizes the relationship between an agent's movement and the environment's geometry.

The Ego2-Map architecture aligns egocentric RGB-D views with top-down semantic maps through specialized ViT encoders and a contrastive loss function.

Contrastive Learning and Semantic Alignment

The model is trained using a dual-stream approach. One stream processes pairs of egocentric RGB-D images captured along a short trajectory, while the other processes a corresponding top-down semantic map. This map provides a distilled version of reality, highlighting obstacles, open spaces, and object categories (e.g., furniture, walls).

By maximizing the similarity between a view pair and its specific map in a shared embedding space, the visual encoder learns to extract structural properties—such as doorway transitions and room layouts—that are typically invisible to standard image-text models. This alignment forces the encoder to internalize the spatial context provided by the map, which then persists even when the map is unavailable during downstream tasks.

The contrastive objective pushes representations of egocentric views ($c^I$) and their corresponding semantic maps ($c^M$) closer together while pushing away unrelated samples.

Auxiliary Spatial Tasks

To further sharpen the model's geometric understanding, the authors incorporate two auxiliary tasks:

  1. Angular Offset Prediction: The model must predict the relative angle between two views taken from the same location, improving its sense of orientation.
  2. Explorable Distance Prediction: The model estimates the distance to the nearest obstacle in its current path, directly aiding in collision avoidance and path planning.

Experimental Setup and Datasets

Pre-training was conducted using the Habitat-Matterport3D (HM3D) dataset, leveraging 800 high-fidelity 3D scans of real homes to generate 500,000 training triplets. The resulting representations were evaluated on two primary benchmarks:

  • R2R-CE (Room-to-Room in Continuous Environments): A Vision-and-Language Navigation (VLN) task requiring agents to follow natural language instructions in a continuous 3D space.
  • ObjNav (Object-Goal Navigation): A task where the agent must locate and navigate to a specific object category in an unfamiliar environment.

Key Results and Insights

Ego2-Map achieved new state-of-the-art results on the R2R-CE test server, reaching a 47% Success Rate (SR) and 41% SPL (Success weighted by Path Length). This represents a significant improvement over previous methods (+3% SR and +4% SPL). In Object-Goal Navigation, Ego2-Map outperformed established models like OVRL and EmbCLIP, despite being trained on a smaller data subset.

A critical finding of the study is the "Frozen Advantage." The visual encoder learned via Ego2-Map can remain frozen during downstream navigation training without sacrificing performance. This suggests that the learned representations are truly generalizable and capture fundamental navigational cues rather than just over-fitting to specific task dynamics.

Conclusion

Ego2-Map demonstrates that semantic maps are powerful supervisors for learning navigational features. By aligning egocentric views with top-down spatial data, the framework effectively "upgrades" standard visual encoders for the 3D world. While the method currently relies on dense semantic annotations for pre-training, it paves the way for more spatially-aware Embodied AI agents capable of navigating complex human environments with greater efficiency and understanding.

Summary updated

Abstract

Being able to perceive the semantics and the spatial structure of the environment is essential for visual navigation of a household robot. However, most existing works only employ visual backbones pre-trained either with independent images for classification or with self-supervised learning methods to adapt to the indoor navigation domain, neglecting the spatial relationships that are essential to the learning of navigation. Inspired by the behavior that humans naturally build semantically and spatially meaningful cognitive maps in their brains during navigation, in this paper, we propose a novel navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps (Ego$^2$-Map). We apply the visual transformer as the backbone encoder and train the model with data collected from the large-scale Habitat-Matterport3D environments. Ego$^2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation. Experiments show that agents using our learned representations on object-goal navigation outperform recent visual pre-training methods. Moreover, our representations significantly improve vision-and-language navigation in continuous environments for both high-level and low-level action spaces, achieving new state-of-the-art results of 47% SR and 41% SPL on the test server.
BibTeX
@inproceedings{hong2023learning,
  author = {Yicong Hong and Yang Zhou and Ruiyi Zhang and Franck Dernoncourt and Trung Bui and Stephen Gould and Hao Tan},
  title = {{Learning Navigational Visual Representations with Semantic Map Supervision}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00284},
}
Figure