Learning words and speech units through natural interactions

Jonas Hörnstein, José Santos-VictorInterspeech 2010 · DOI

Introduction

Traditional Automatic Speech Recognition (ASR) systems typically rely on massive, hand-labeled datasets and predefined phoneme models. This "top-down" approach contrasts sharply with human language acquisition. Infants learn to communicate through sparse, noisy, and multimodal interactions with caregivers, often developing linguistic understanding before they can produce clear speech. This paper explores an "ecological" alternative: a robot that learns speech units and word meanings through natural interactions, mimicking the developmental stages of a human infant.

Methodology

The researchers implemented a three-stage developmental pipeline on a humanoid robot to facilitate unsupervised language learning.

1. Multimodal Word Learning

The robot processes a continuous audio stream alongside its visual field. It utilizes a short-term memory buffer (10–20 seconds) to identify recurring acoustic patterns that co-occur with specific visual objects, represented by their silhouettes. By applying Mutual Information (MI), the system filters out environmental noise and establishes consistent associations between sounds and shapes. This hierarchical clustering allows the robot to map acoustic word candidates to physical objects.

Hierarchical clustering of acoustic word candidates and visual object features.

2. Motor-Based Bootstrapping

Inspired by the "Motor Theory of Speech Perception," the authors posit that the ability to produce a sound enhances the ability to recognize it. The robot, equipped with a simulated vocal tract, first engages in "babbling" to map motor positions to acoustic outputs. It then participates in "imitation games" with a caregiver, receiving positive reinforcement when it successfully mimics specific vowels.

3. Unsupervised Statistical Modeling

The robot constructs Hidden Markov Models (HMMs) for words. Rather than random initialization, the system uses speech units learned during imitation as "seeds" for K-means clustering. The feature vector for these models is a 46-dimensional hybrid, combining traditional Mel-frequency cepstral coefficients (MFCCs) with the robot's own internal motor-tract positions.

Key Insights

  • Infant-Directed Speech (IDS): The system leverages natural human cues, such as focal stress and utterance-final positioning, to isolate word candidates.
  • Motor-Acoustic Synergy: Including articulatory data in the feature vector provides a more robust representation of sounds.
  • Emergent Phonemes: Phoneme-like units emerge naturally as the system seeks to efficiently distinguish between an increasing vocabulary, rather than being pre-programmed.

Key Results

The system demonstrated high efficacy in learning from real-time interactions. It achieved a peak 98% word recognition rate when configured with 10 speech units and utilizing motor-based bootstrapping.

Word recognition accuracy comparing bootstrapped vs. non-bootstrapped models.

As shown in the performance data, the "bootstrapped" models consistently outperformed the "no bootstrap" models, especially when the number of speech units was low (5–8 units). Furthermore, the unsupervised speech units often developed a near one-to-one mapping with real phonemes (e.g., internal unit "1" corresponding to /a/).

Conclusion

This research demonstrates that robots can bootstrap linguistic knowledge through multimodal interaction and motor imitation. While the current study focused on a limited vocabulary of eight words in a controlled environment, it provides a foundation for scaling toward more complex linguistic structures, including actions, events, and grammar. Future work will investigate the system's performance with larger datasets and more diverse acoustic environments.

Summary updated

BibTeX
@inproceedings{hornstein2010learning,
  author = {Jonas Hörnstein and José Santos-Victor},
  title = {{Learning words and speech units through natural interactions}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-191},
}
Figure