MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation

Zachary Seymour, Kowshik Thopalli, Niluthpol Chowdhury Mithun, Han-Pang Chiu, Supun Samarasekera, Rakesh Kumar

Abstract

Visual navigation for autonomous agents is a core task in computer vision and robotics. While learning-based methods like deep reinforcement learning (DRL) show potential, they often require massive computational resources. We propose MaAST (Map Attention with Semantic Transformers), a novel approach that encodes vital scene semantics—such as traversable paths, unexplored areas, and observed objects—into a semantically informed, top-down egocentric map. By introducing a 2D map attention mechanism based on multi-layer Transformer networks, we enable agents to navigate efficiently under a strict computational budget. Experiments on 3D reconstructed indoor environments demonstrate that MaAST outperforms multiple baselines while operating with an 80% decrease in required training experience.

Introduction

Deep Reinforcement Learning (DRL) has advanced autonomous navigation, yet models remain notoriously "data-hungry," often requiring hundreds of millions of interactions to learn basic policies. While humans use semantic cues (e.g., "head toward the door") to navigate, most DRL agents rely on raw visual streams or simple occupancy maps that lack high-level nuance. Furthermore, standard convolutional neural networks (CNNs) often struggle to extract relevant information from dense semantic data. MaAST addresses these challenges by providing a structured framework for agents to build and intelligently "attend" to semantic maps.

The MaAST Framework

The MaAST architecture consists of three primary components designed to maximize sample efficiency and navigational intelligence:

  1. Egocentric Semantic Map Construction: As the agent moves, it projects 2D semantic segmentation and depth observations into a 3D point cloud, which is then flattened into a top-down, egocentric 2D map. Each grid cell stores a "bag-of-classes" binary vector representing present objects.
  2. The Map Transformer: Instead of a standard CNN, MaAST utilizes a Transformer architecture adapted for spatial navigation. This includes a novel 2D Map Attention mechanism with specialized positional embeddings. A Gaussian kernel centered on the agent helps the model prioritize objects based on their spatial proximity.
  3. Reinforcement Learning with Auxiliary Rewards: The model is trained using Proximal Policy Optimization (PPO). To accelerate learning, an auxiliary reward based on the percentage of the map revealed is added, encouraging active exploration.
Figure 1: The MaAST architecture showing the flow from multi-modal inputs at time t to action selection at time t+1.

Experimental Setup and Data Modalities

We evaluated MaAST using the Habitat simulator on the Matterport3D dataset, which features 90 high-resolution 3D reconstructions of real-world houses. The agent processes multiple input streams simultaneously to build its internal representation of the environment.

Figure 2: Visualization of the input modalities (RGB, Depth, Semantic Segmentation) and the resulting egocentric semantic maps used by the agent.

Key Results and Insights

  • Superior Efficiency: MaAST achieved a 47% Success weighted by Path Length (SPL), significantly outperforming the standard RGB-D baseline (34%) and the RGB-D + Occupancy map baseline (43%).
  • Attention is Essential: A critical finding was that simply adding semantic information is insufficient; naive CNN-based approaches often performed worse than simple occupancy maps. The Transformer-based attention mechanism is vital for filtering noise and focusing on relevant landmarks.
  • Sample Efficiency: By providing a superior memory structure and "reader," the agent reaches high performance in just 14 million steps—a fraction of the 75 million+ steps typically required by competing methods.
  • Robustness: The system remains effective even when using noisy, predicted semantic masks (via Mask R-CNN) rather than ground-truth labels.

Conclusion and Future Work

MaAST demonstrates that combining rich semantic memory with Transformer-based spatial attention allows for highly efficient visual navigation. While the current model assumes perfect localization, future work will explore handling pose estimation errors (SLAM drift) and navigating dynamic environments with moving obstacles. Additionally, integrating MaAST into hierarchical policies could further enhance performance in complex, multi-floor environments.

Summary updated

Abstract

Visual navigation for autonomous agents is a core task in the fields of computer vision and robotics. Learning-based methods, such as deep reinforcement learning, have the potential to outperform the classical solutions developed for this task; however, they come at a significantly increased computational load. Through this work, we design a novel approach that focuses on performing better or comparable to the existing learning-based solutions but under a clear time/computational budget. To this end, we propose a method to encode vital scene semantics such as traversable paths, unexplored areas, and observed scene objects -- alongside raw visual streams such as RGB, depth, and semantic segmentation masks -- into a semantically informed, top-down egocentric map representation. Further, to enable the effective use of this information, we introduce a novel 2-D map attention mechanism, based on the successful multi-layer Transformer networks. We conduct experiments on 3-D reconstructed indoor PointGoal visual navigation and demonstrate the effectiveness of our approach. We show that by using our novel attention schema and auxiliary rewards to better utilize scene semantics, we outperform multiple baselines trained with only raw inputs or implicit semantic information while operating with an 80% decrease in the agent's experience.
BibTeX
@inproceedings{seymour2021maast,
  author = {Zachary Seymour and Kowshik Thopalli and Niluthpol Chowdhury Mithun and Han-Pang Chiu and Supun Samarasekera and Rakesh Kumar},
  title = {{MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=SSgWhAAtKJ5},
}
Figure