Unsupervised Lexical Acquisition of Relative Spatial Concepts Using Spoken User Utterances

Rikunari Sagara, Ryo Taguchi, Akira Taniguchi, Tadahiro Taniguchi, Koosuke Hattori, Masahiro Hoguro, Taizo Umezaki

Abstract

This paper presents ReSCAM (Relative Spatial Concept Acquisition Method), a probabilistic framework that enables robots to learn relative spatial concepts (e.g., "front", "right") and their corresponding words directly from raw spoken human utterances. By treating the reference object as a hidden variable within a Bayesian generative model, the system can simultaneously perform word segmentation from audio and map these words to spatial distributions. This approach allows robots to acquire language without prior linguistic knowledge, even when speakers omit explicit references to objects. Experimental results show that the model effectively clusters spatial concepts and scales to complex environments when linguistic clues are available.

Introduction

For service robots to interact naturally with humans, they must understand spatial language within their specific environments. While previous research has focused on "absolute" concepts (e.g., "this is the kitchen"), relative spatial concepts are significantly more complex. The meaning of a word like "right" is entirely dependent on a reference object—for instance, is it to the right of the TV, the chair, or the robot itself?

Illustration of the ambiguity in relative spatial concepts where the reference object (e.g., chair or TV) must be inferred.

In natural human speech, the reference object is frequently omitted or referred to using unknown vocabulary. This creates a "lexical acquisition" challenge: the robot must segment a continuous stream of phonemes into words while simultaneously identifying which object in the room serves as the anchor for the spatial description.

Methods

The proposed solution, ReSCAM, utilizes a Bayesian generative model to solve the joint problem of word segmentation and spatial mapping. The framework treats the reference object and the spatial concept as latent variables to be inferred from the data.

ReSCAM and ReSCAM+O

  1. ReSCAM: Designed for scenarios where the reference object is not named. It uses a Chinese Restaurant Process (CRP) to dynamically estimate the number of spatial concepts and a von Mises distribution to model the angles of these concepts relative to candidate objects. Inference is performed using Metropolis-Hastings (MCMC) sampling.
  2. ReSCAM+O: An extension that leverages linguistic "clues." If a user mentions an object (e.g., "right of the chair"), the model uses object recognition results and word co-occurrence to narrow the search space for the reference object.
The probabilistic graphical model for ReSCAM, integrating spatial distributions and word segmentation.

Technical Innovations

To handle the inherent noise in speech recognition, the system employs Mutual Information (MI) Selection. It generates multiple candidate word sequences and selects the one that maximizes the correlation between the words and the observed physical locations. Additionally, class n-grams are used to categorize potential location words, improving the probability of correct segmentation.

Experimental Evaluation

The methods were evaluated using a dataset of 76 Japanese utterances (covering 4 spatial concepts) and simulated 2D coordinate data relative to various household objects. The Julius engine provided phoneme recognition, while latticelm was used for unsupervised segmentation.

Key Results

  • Clustering Accuracy: In simple environments with three candidate objects, ReSCAM achieved an Adjusted Rand Index (ARI) of nearly 1.0, indicating near-perfect spatial concept clustering.
  • Lexical Discovery: The system successfully extracted phoneme sequences for directional terms such as "mae" (front), "ushiro" (back), "migi" (right), and "hidari" (left) from continuous speech.
  • Scalability: While the basic ReSCAM model's performance declined as the number of objects increased to 20, ReSCAM+O maintained high accuracy. This demonstrates that even minimal linguistic or visual clues are vital for scaling unsupervised learning to real-world complexity.

Conclusion and Future Work

ReSCAM demonstrates that spatial consistency can be used to bootstrap language learning, and linguistic patterns can conversely refine spatial understanding. However, the current model assumes a fixed perspective (the robot's). Future work will explore perspective ambiguity, where the system must estimate whether the speaker is using their own or the robot's coordinate system. Additionally, the authors aim to extend the framework to non-directional relative concepts, such as "larger than" or "brighter than."

Summary updated

BibTeX
@article{sagara2022unsupervised,
  author = {Rikunari Sagara and Ryo Taguchi and Akira Taniguchi and Tadahiro Taniguchi and Koosuke Hattori and Masahiro Hoguro and Taizo Umezaki},
  title = {{Unsupervised Lexical Acquisition of Relative Spatial Concepts Using Spoken User Utterances}},
  journal = {Adv. Robotics},
  year = {2022},
  doi = {10.1080/01691864.2021.2007168},
}
Figure