Recognizing speech in a novel accent: the motor theory of speech perception reframed

Clément Moulin-Frier, Michael A. Arbib

This paper, published in Biological Cybernetics, explores the intersection of linguistics, neuroscience, and computational modeling to answer a fundamental question: How do our brains adapt so quickly to a foreign accent?

TL;DR

The authors propose a dual-stream neuro-computational framework that reframes the "Motor Theory of Speech Perception." By using a Hidden Markov Model (HMM) to simulate how listeners map unfamiliar sounds to their own native phonemes, they demonstrate that we can "bootstrap" recognition of a novel accent using just a few dozen words of lexical feedback.

What Problem Does This Paper Address?

For decades, the Motor Theory of Speech Perception has argued that we understand speech by internally simulating the motor gestures (tongue and lip movements) required to produce those sounds. However, this theory struggles to explain how we understand someone with a completely novel foreign accent. If we haven't learned their specific "motor gestures" yet, how do we decode their speech?

The authors address the gap between purely auditory perception and motor-driven simulation. They seek to explain the "Aha!" moment when, after a few sentences of struggling with a new accent, the listener suddenly begins to understand the speaker with ease.

Methods

The researchers developed a novel computational model based on Hidden Markov Models (HMMs).

  1. Representation: They represented speech sounds (phones) as feature vectors based on International Phonetic Association (IPA) standards (e.g., place of articulation, manner, voicing).
  2. The Model:
    • Hidden States: The listener’s native phonemes.
  3. Observations: The actual sounds (phones) produced by the foreign speaker.
  4. Transitions: Probabilities of phoneme sequences in the native language.
  5. Adaptation Algorithm: The model starts with "naïve" recognition. When it encounters a word it can identify (using context or "lexical feedback"), it uses the Viterbi algorithm to find the most likely sequence of insertions, deletions, or substitutions the speaker made.
  6. Learning: It then uses a Laplace succession rule to update its internal "conversion table," refining the mapping between the speaker's unique sounds and the listener's native phonemes.

Datasets

  • Speech Accent Archive (Weinberger 2010): A corpus of phonetic transcriptions of a standard "elicitation paragraph" ("Please call Stella...") spoken by hundreds of people with different native languages.
  • MRC Psycholinguistic Database: Used to establish a "native" English baseline, providing standard transcriptions and word frequencies for approximately 10,000 words.

Key Insights and Ideas

  • The "Agnostic" Model: The authors show that rapid adaptation can be achieved through a purely statistical mapping process. This model works whether the underlying representations are auditory, motoric, or abstract.
  • Lexical Feedback is Key: We don't learn an accent by studying sounds in isolation; we use words we already recognize to reverse-engineer the speaker's "code."
  • Dual-Stream Reframing: The authors propose that speech perception isn't a single process.
  • The Ventral Stream (Auditory-to-Meaning) handles most everyday speech.
  • The Dorsal Stream (Auditory-to-Motor/Mirror System) acts as a "helper" that kicks in during difficult conditions—like heavy noise, non-words, or learning a novel accent.
  • Mirror Neurons as Parity Builders: Instead of being the primary engine of perception, the mirror neuron system is framed as a mechanism for maintaining "parity" (a common ground) between how we hear and how we speak.

Key Results

  • Rapid Improvement: In a simulation of a French speaker's accent, the model's word recognition rate jumped from 62% (naïve) to 97% after "hearing" only the first 35 words of the paragraph.
  • Statistical Significance: Using a two-way ANOVA, the researchers found that their learning process was significantly more effective for foreign speakers than for native speakers (who already align closely with the listener's internal model).
  • Generalization: The model successfully applied what it learned from a small "training set" of words to correctly identify completely new words in the "test set" spoken in the same accent.

Limitations and Future Work

  • Hand-Coded Data: The model currently operates on phonetic transcriptions (text-like symbols) rather than raw acoustic signals (spectrograms). Real-world audio contains much more "noise" and variability.
  • Coarticulation: The model treats speech as a sequence of discrete units. In reality, sounds overlap and influence each other (coarticulation), which the current HMM does not account for.
  • Vocabulary Size: The experiments were limited to a small 69-word paragraph. Future work needs to test if this rapid "bootstrapping" scales to natural, large-vocabulary conversations.
  • Neuro-Validation: While the dual-stream architecture is biologically inspired, more fMRI or EEG studies are needed to confirm if the brain actually shifts load between these streams during accent adaptation as the model predicts.

Summary updated

Abstract

The motor theory of speech perception holds that we perceive the speech of another in terms of a motor representation of that speech. However, when we have learned to recognize a foreign accent, it seems plausible that recognition of a word rarely involves reconstruction of the speech gestures of the speaker rather than the listener. To better assess the motor theory and this observation, we proceed in three stages. Part 1 places the motor theory of speech perception in a larger framework based on our earlier models of the adaptive formation of mirror neurons for grasping, and for viewing extensions of that mirror system as part of a larger system for neuro-linguistic processing, augmented by the present consideration of recognizing speech in a novel accent. Part 2 then offers a novel computational model of how a listener comes to understand the speech of someone speaking the listener’s native language with a foreign accent. The core tenet of the model is that the listener uses hypotheses about the word the speaker is currently uttering to update probabilities linking the sound produced by the speaker to phonemes in the native language repertoire of the listener. This, on average, improves the recognition of later words. This model is neutral regarding the nature of the representations it uses (motor vs. auditory). It serve as a reference point for the discussion in Part 3, which proposes a dual-stream neuro-linguistic architecture to revisits claims for and against the motor theory of speech perception and the relevance of mirror neurons, and extracts some implications for the reframing of the motor theory.
BibTeX
@inproceedings{moulinfrier2013recognizing,
  author = {Clément Moulin-Frier and Michael A. Arbib},
  title = {{Recognizing speech in a novel accent: the motor theory of speech perception reframed}},
  booktitle = {Biol. Cybern},
  year = {2013},
  url = {https://openreview.net/forum?id=eSBL1i7XSHO},
}
Figure