RSMPNet: Relationship Guided Semantic Map Prediction

Jingwen Sun, Jing Wu, Ze Ji, Yu-Kun LaiWACV 2024 · OpenReview

Abstract

In semantic navigation, a top-down map with accurate and complete semantic information is vital to subsequent decision-making. However, due to occlusions and limitations of the robot’s field of view (FOV), there are often unobserved areas in the top-down maps. To address this problem, recent works have studied semantic map prediction to complete the top-down maps. In this work, we propose to improve map prediction by integrating relational information. We propose RSMPNet, a relationship-guided semantic map prediction network, which makes use of semantic and spatial relationships to predict unobserved areas from accumulated semantic maps. Specifically, we propose a Relationship Reasoning Layer that includes two modules, namely 1) the Semantic Relationship Graph Reasoning Module (SeGRM) to capture the semantic relationship and 2) the Spatial Relationship Graph Reasoning Module (SpGRM) to utilize the spatial relationship. We also design a semantic relationship enhanced loss to enhance our model to learn semantic relationship information. Experiments show the effectiveness of our proposed network which achieves state-of-the-art performance in semantic map prediction.

Introduction

When a robot navigates an indoor environment, its internal "top-down" map is often incomplete due to a limited field of view (FOV) and physical occlusions. These "unknown" regions hinder effective decision-making and navigation. While previous research has focused on geometric data or simple object categories for map completion, they often overlook the contextual relationships inherent in human environments. For instance, a robot seeing a stove can reasonably infer that a sink might be nearby.

Overview of the semantic map prediction task, showing the transition from partial observations to a complete semantic map.

RSMPNet addresses this by explicitly modeling semantic and spatial relationships as priors to "fill in the blanks" of unobserved areas with higher accuracy than previous methods.

Methodology

The proposed RSMPNet (Relationship-guided Semantic Map Prediction Network) utilizes a UNet-based architecture enhanced with a specialized Relationship Reasoning Layer (RRL).

The RSMPNet architecture featuring the Relationship Reasoning Layer (RRL) with parallel SeGRM and SpGRM modules.

Relationship Reasoning Layer (RRL)

The RRL employs Graph Convolutional Networks (GCNs) to process map features through two parallel modules:

  1. Semantic Relationship Graph Reasoning Module (SeGRM): This module captures semantic similarities, allowing the network to associate pixels of related categories (e.g., different types of furniture) regardless of their spatial distance.
  2. Spatial Relationship Graph Reasoning Module (SpGRM): This module leverages spatial proximity, using Manhattan distance to ensure that nearby pixels strongly influence a specific pixel's representation, reflecting the natural clustering of indoor objects like chairs around a table.

The authors found that a parallel configuration of these modules prevents one type of relationship from dominating the other, leading to superior feature fusion. Additionally, a Semantic Relationship Enhanced Loss function is introduced to align the model's learned relationship patterns with ground-truth environmental structures.

Experimental Results

The model was evaluated using the Matterport3D (MP3D) dataset within the Habitat simulator, utilizing a large-scale dataset of over 39,000 accumulated maps to reflect realistic robotic navigation scenarios.

Quantitative Performance

RSMPNet achieved state-of-the-art results across standard metrics:

  • mIoU (Mean Intersection over Union): 26.33% (compared to a 24.87% baseline).
  • mAcc (Mean Pixel Accuracy): 35.21% (compared to a 33.18% baseline).

Qualitative Analysis

Visually, RSMPNet demonstrates a superior ability to predict logical object clusters, such as stools around a counter, while maintaining the integrity of already observed objects—a task where previous models often struggled with blurring or misidentification.

Conclusion

RSMPNet demonstrates that spatial and semantic relationships are powerful priors for dense map prediction. By treating individual pixels as nodes in a relational graph, the network achieves fine-grained completion of unobserved areas. Future work will explore adaptive integration of reasoning modules across different network layers and balancing these priors to better handle unusual or "outlier" object placements.

Summary updated

Abstract

In semantic navigation, a top-down map with accurate and complete semantic information is vital to subsequent decision-making. However, due to occlusions and limitations of the robot’s field of view (FOV), there are often unobserved areas in the top-down maps. To address this problem, recent works have studied semantic map prediction to complete the top-down maps. In this work, we propose to improve map prediction by integrating relational information. We propose RSMPNet, a relationship-guided semantic map prediction network, which makes use of semantic and spatial relationships to predict unobserved areas from accumulated semantic maps. Specifically, we propose a Relationship Reasoning Layer that includes two modules, namely 1) the Semantic Relationship Graph Reasoning Module (SeGRM) to capture the semantic relationship and 2) the Spatial Relationship Graph Reasoning Module (SpGRM) to utilize the spatial relationship. We also design a semantic relationship enhanced loss to enhance our model to learn semantic relationship information. Experiments show the effectiveness of our proposed network which achieves state-of-the-art performance in semantic map prediction. Our code and dataset are publicly available at https://github.com/jws39/semantic-map-prediction
BibTeX
@inproceedings{sun2024rsmpnet,
  author = {Jingwen Sun and Jing Wu and Ze Ji and Yu-Kun Lai},
  title = {{RSMPNet: Relationship Guided Semantic Map Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=3SK0wNUj2O},
}
Figure