Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping

Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro Taniguchi, Tetsunari Inamura

Abstract

In this paper, we propose an online learning algorithm based on a Rao-Blackwellized particle filter for spatial concept acquisition and mapping. We have proposed a nonparametric Bayesian spatial concept acquisition model (SpCoA). We propose a novel method (SpCoSLAM) integrating SpCoA and FastSLAM in the theoretical framework of the Bayesian generative model. The proposed method can simultaneously learn place categories and lexicons while incrementally generating an environmental map. Furthermore, the proposed method has scene image features and a language model added to SpCoA. In the experiments, we tested online learning of spatial concepts and environmental maps in a novel environment of which the robot did not have a map. Then, we evaluated the results of online learning of spatial concepts and lexical acquisition. The experimental results demonstrated that the robot was able to more accurately learn the relationships between words and the place in the environmental map incrementally by using the proposed method.

Introduction

Autonomous robots operating in human environments face a significant "chicken-and-egg" challenge: to understand semantic commands like "Go to the kitchen," a robot typically requires a pre-existing map with labeled regions. However, building such a map is most effective when the robot already understands the semantic nature of the spaces it explores. Traditionally, spatial concept acquisition—learning what a "kitchen" or "corridor" is—has been treated as a batch-processing task, where data is collected first and processed offline.

Overview of the SpCoSLAM online learning process, showing the robot acquiring spatial concepts and place names while mapping.

This paper introduces SpCoSLAM, an online algorithm that enables a robot to simultaneously build a geometric map of a new environment while learning the names and categories of the places within it. By integrating spatial concept acquisition with Simultaneous Localization and Mapping (SLAM) into a single Bayesian generative model, the robot can learn vocabulary and spatial layouts on the fly from human speech and sensor data without any prior knowledge.

Methodology: The SpCoSLAM Framework

The core of the proposed method is the integration of two distinct frameworks: SpCoA (Spatial Concept Acquisition) and FastSLAM. SpCoA is a nonparametric Bayesian model designed to cluster multimodal data—including position, image features, and speech—into coherent "spatial concepts." FastSLAM is a well-established robotics algorithm used for mapping and localization.

Technical Implementation

  • Rao-Blackwellized Particle Filter (RBPF): SpCoSLAM utilizes an RBPF where each particle represents a specific hypothesis of the robot's trajectory, the environmental map, and the learned spatial concepts. This allows the system to maintain multiple potential interpretations of the environment simultaneously.
  • Multimodal Data Processing: The system integrates three primary data streams: laser scans for occupancy grid maps; computer vision using a pre-trained Places205-AlexNet to extract high-level features from camera images; and speech recognition where raw audio is processed through the Julius engine and segmented into words using latticelm, an unsupervised word segmentation method.
  • Sequential Language Model Updates: A key innovation in SpCoSLAM is the continuous update of its internal language model. As the robot hears more speech, it refines its dictionary, allowing for better recognition of recurring place names over time.

Experimental Evaluation

The researchers validated SpCoSLAM using the "albert-b-laser-vision" dataset from the Radish repository. To simulate human interaction, they recorded 50 Japanese speech utterances corresponding to the robot's movements, covering nine distinct place names such as "Sharing desk" and "Printer room."

Incremental acquisition of spatial concepts and lexicons at different steps (15, 30, and 50) during exploration.

Key Results and Insights

  • Mutual Complementarity: The experiments highlighted that spatial and linguistic information are mutually reinforcing. A robot's belief that it is in a "corridor" helps it correctly identify the word "corridor" even if the audio is noisy. Conversely, hearing a recognized place name helps the robot refine its estimated position on the map.
  • Improved Word Discovery: By updating the language model sequentially, SpCoSLAM effectively addresses the problem of "over-segmentation," where unsupervised systems mistakenly break single words into multiple meaningless fragments.
  • Quantitative Performance: SpCoSLAM outperformed baseline models in Normalized Mutual Information (NMI) for position distributions and achieved a higher Place Recognition Rate (PRR). This indicates that the robot's semantic understanding of the environment improved significantly as exploration progressed.

Conclusion and Future Directions

SpCoSLAM represents a significant step toward truly autonomous robots that can learn from their surroundings and human teachers in real-time. The system successfully demonstrates that a robot can start with a blank map and zero vocabulary, building both incrementally through exploration.

Future research will address the challenges of scaling this approach to larger, more complex buildings. Additionally, the authors suggest implementing "forgetting" and "rejuvenation" mechanisms to handle dynamic environments where layouts or room functions might change over time. Testing the system with more linguistically diverse datasets will also be crucial for validating the robustness of the unsupervised word segmentation approach.

Summary updated

Abstract

In this paper, we propose an online learning algorithm based on a Rao-Blackwellized particle filter for spatial concept acquisition and mapping. We have proposed a nonparametric Bayesian spatial concept acquisition model (SpCoA). We propose a novel method (SpCoSLAM) integrating SpCoA and FastSLAM in the theoretical framework of the Bayesian generative model. The proposed method can simultaneously learn place categories and lexicons while incrementally generating an environmental map. Furthermore, the proposed method has scene image features and a language model added to SpCoA. In the experiments, we tested online learning of spatial concepts and environmental maps in a novel environment of which the robot did not have a map. Then, we evaluated the results of online learning of spatial concepts and lexical acquisition. The experimental results demonstrated that the robot was able to more accurately learn the relationships between words and the place in the environmental map incrementally by using the proposed method.
BibTeX
@inproceedings{taniguchi2017online,
  author = {Akira Taniguchi and Yoshinobu Hagiwara and Tadahiro Taniguchi and Tetsunari Inamura},
  title = {{Online Spatial Concept and Lexical Acquisition with Simultaneous Localization and Mapping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=q1MWsPpLJW2},
}
Figure