Semantic Mapping Based on Spatial Concepts for Grounding Words Related to Places in Daily Environments

Yuki Katsumata, Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro TaniguchiFrontiers Robotics AI 2019 · DOI · OpenReview

Abstract

Autonomous service robots operating in home environments must understand the semantic layout of their surroundings to execute human commands effectively. This paper introduces SpCoMapping, a novel statistical framework that enables robots to learn the names and physical boundaries of rooms through unsupervised, multimodal learning. By combining spatial concept acquisition with Markov Random Fields (MRFs), the system allows robots to discover complex, arbitrary room shapes using raw sensor data and natural user utterances, bypassing the need for predefined place names.

Introduction

For a service robot to follow a command like "clean Joseph's room," it must solve two fundamental challenges: grounding (linking the linguistic term to a specific location) and segmentation (defining the physical extent of that location). Traditional semantic mapping methods often struggle with the "overwrite problem," where new observations erase previous statistical data, or they rely on rigid geometric assumptions (like ellipses) that fail to capture the actual layout of human homes. Furthermore, many systems use closed vocabularies, preventing them from adapting to user-specific terminology.

Methods

SpCoMapping addresses these issues by integrating probabilistic spatial concept acquisition with a Markov Random Field (MRF). The framework fuses three distinct data streams: visual data from a CNN, spatial data from self-localization, and linguistic data from user utterances.

Overview of the SpCoMapping framework showing multimodal data integration and the resulting semantic map.

Unlike previous models that used Gaussian distributions to represent areas, the MRF models dependencies between neighboring pixels on an occupancy grid. This allows the robot to learn arbitrary shapes that respect physical boundaries like walls. Using a Dirichlet Process (DP) prior, the robot can automatically estimate the number of distinct categories (rooms) without prior knowledge. Inference is performed using Gibbs sampling to iteratively refine the semantic map, word distributions, and visual features.

Experimental Setup and Datasets

The system was evaluated in both simulated and real-world environments. Simulation experiments utilized SIGVerse, testing across five distinct house layouts. Real-world data was collected using a Toyota Human Support Robot (HSR) equipped with an RGB-D camera and laser range sensor in a laboratory living space. The CNN component utilized the Places205 dataset for initial scene recognition capabilities.

Key Results

The researchers compared SpCoMapping against several baselines using the Adjusted Rand Index (ARI) to measure clustering accuracy.

  • Clustering Performance: SpCoMapping (DP) achieved an average ARI of 0.5066 in simulation, significantly outperforming a CNN-based baseline (0.2795) and previous Gaussian-based spatial concept methods (0.3499).
  • Qualitative Success: In real-world deployments, the robot successfully generated semantic maps where regions like "kitchen" and "meeting space" were correctly bounded by the room's physical geometry.
  • Word Grounding: The system demonstrated high accuracy in locating rooms when prompted with specific words by calculating the highest probability region associated with those vocabulary features.

Discussion and Future Work

SpCoMapping represents a step toward "symbol emergence," where robots ground language in the physical world through consistent co-occurrence of visual patterns, coordinates, and user utterances. By treating walls as hard constraints within the MRF, the model avoids the "bleeding" effect common in other semantic mapping techniques.

However, several limitations remain. The current implementation is a batch process; future work aims to develop an online learning version integrated with SLAM for real-time adaptation. Additionally, extending the model from 2D to 3D semantic space will be crucial for complex object manipulation tasks. Finally, improving natural language processing to filter out linguistic noise will enhance the robustness of word grounding.

Summary updated

Abstract

An autonomous robot performing tasks in a human environment needs to recognize semantic information about places. Semantic mapping is a task in which suitable semantic information is assigned to an environmental map so that a robot can communicate with people and appropriately perform tasks requested by its users. We propose a novel statistical semantic mapping method called SpCoMapping, which integrates probabilistic spatial concept acquisition based on multimodal sensor information and a Markov random field applied for learning the arbitrary shape of a place on a map.SpCoMapping can connect multiple words to a place in a semantic mapping process using user utterances without pre-setting the list of place names. We also develop a nonparametric Bayesian extension of SpCoMapping that can automatically estimate an adequate number of categories. In the experiment in the simulation environments, we showed that the proposed method generated better semantic maps than previous semantic mapping methods; our semantic maps have categories and shapes similar to the ground truth provided by the user. In addition, we showed that SpCoMapping could generate appropriate semantic maps in a real-world environment.
BibTeX
@article{katsumata2019semantic,
  author = {Yuki Katsumata and Akira Taniguchi and Yoshinobu Hagiwara and Tadahiro Taniguchi},
  title = {{Semantic Mapping Based on Spatial Concepts for Grounding Words Related to Places in Daily Environments}},
  journal = {Frontiers Robotics AI},
  year = {2019},
  doi = {10.3389/frobt.2019.00031},
}
Figure