A computational model of perceptuo-motor processing in speech perception: learning to imitate and categorize synthetic CV syllables

Raphaël Laurent, Jean-Luc Schwartz, Pierre Bessière, Julien DiardInterspeech 2013

TL;DR

This paper introduces COSMO, a Bayesian computational model that integrates auditory and motor information for speech perception. By learning to imitate synthetic speech sounds, COSMO demonstrates that motor knowledge enhances robustness to noise and improves syllable categorization, suggesting a unified approach to understanding how humans learn and process speech.

What Problem Does This Paper Address?

The paper tackles a fundamental question in speech perception: how do humans integrate auditory and motor information to understand speech, and what is the role of motor information in this process? While it's widely accepted that motor information plays a role, two key questions remain: what is the nature of this information, and how do listeners learn the link between motor commands and perceived speech? Existing research has explored this integration, but often lacks a clear explanation of why motor information is beneficial or a realistic model of how humans learn this perceptuo-motor link. This paper aims to bridge this gap by proposing a computational model that learns this link through imitation.

Methods

The core of the paper is the COSMO (Communicating Objects using SensoriMotor Operations) model, a Bayesian framework designed to unify speech production, perception, and imitation. For speech perception, COSMO focuses on inferring the probability of a message (syllable) given sensory input.

The model utilizes a realistic Variable Linear Articulatory Model (VLAM) to generate synthetic CV (consonant-vowel) syllables. VLAM uses seven articulatory parameters to model the human vocal tract and compute acoustic features (formants). CV syllables are generated by defining vowel prototypes and modeling consonants as local perturbations of these vowel gestures.

COSMO's learning process is structured into three developmental stages:

  1. Auditory Categorization: Learning the direct mapping from acoustic features to syllable labels using supervised learning.
  2. Sensory-Motor Link Learning (Babbling/Imitation): This is a key novel contribution. The model learns the articulatory-to-acoustic mapping without supervision by attempting to imitate target acoustic syllables. It infers motor gestures that would produce the target sounds, uses VLAM to simulate the acoustic output, and updates its internal model based on this simulated feedback. This is described as "target-oriented babbling."
  3. Motor Categorization: Learning the mapping from motor gestures to syllable labels, leveraging the learned sensory-motor link.

The paper then evaluates three instantiations of COSMO: purely auditory, purely motor, and perceptuo-motor, comparing their performance in syllable categorization under varying levels of noise.

Datasets

The paper does not use pre-existing datasets in the traditional sense. Instead, it generates its own synthetic dataset of 9 CV syllables (/ba/, /bi/, /bu/, /ga/, /gi/, /gu/, /da/, /di/, /du/) using the VLAM articulatory model. These synthetic syllables are characterized by their acoustic features (formants F1, F2, F3). This synthetic data is used for both training the model's different components and for evaluating its performance.

Key Insights and Ideas

  • Motor knowledge enhances robustness to noise: The study strongly suggests that incorporating motor information makes speech perception systems more resilient to noisy acoustic environments.
  • Imitation as a learning mechanism: The "target-oriented babbling" approach demonstrates a plausible developmental scenario for how humans might learn the complex link between motor commands and acoustic speech signals without explicit supervision.
  • Perceptuo-motor fusion outperforms individual modalities: The combined perceptuo-motor model consistently performs better than purely auditory or purely motor models, highlighting the synergistic benefits of integrating both types of information.
  • Motor knowledge complements auditory knowledge: Motor information doesn't just add robustness; it actively enhances the perception process by providing complementary cues.

Key Results

The paper presents simulation results comparing the correct recognition rates of the auditory, motor, and perceptuo-motor models across different levels of Gaussian evaluation noise.

  • No Noise: The purely auditory model performs best when there is no noise.
  • With Noise: As soon as noise is introduced, the motor model becomes more accurate than the auditory model.
  • Perceptuo-Motor: The perceptuo-motor model consistently outperforms both the auditory and motor models across all noise levels.
  • The authors conclude that the motor model's good performance validates their learning-by-imitation algorithm.

Limitations and Future Work

  • Supervised Learning for Categorization: The current version of COSMO relies on supervised learning for auditory and motor categorization, requiring explicit syllable labels. The authors acknowledge this is unrealistic and plan to explore co-constructing motor and auditory systems in parallel.
  • Vocal Tract Growth: The VLAM model does not account for vocal tract growth during development. Future work could incorporate this to simulate developmental changes.
  • Co-construction of Systems: The authors explicitly state that future work will focus on exploring different learning paradigms where motor and auditory systems are co-constructed in parallel, aligning with the Perception-for-Action-Control Theory (PACT).

Summary updated

BibTeX
@inproceedings{laurent2013computational,
  author = {Raphaël Laurent and Jean-Luc Schwartz and Pierre Bessière and Julien Diard},
  title = {{A computational model of perceptuo-motor processing in speech perception: learning to imitate and categorize synthetic CV syllables}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure