Computational modelling of the recognition of foreign-accented speech

Odette Scharenborg, Marijt Witteman, Andrea WeberInterspeech 2012

TL;DR

This paper explores how well a computational model, Fine-Tracker, can simulate human listeners' ability to recognize foreign-accented speech. The model successfully captures some general trends, like increased difficulty with stronger accents, but struggles to replicate individual word recognition patterns, suggesting that current modeling approaches may need refinement to fully account for accent adaptation.

What Problem Does This Paper Address?

Understanding spoken language is usually effortless, but foreign accents introduce variability that can make comprehension harder. While human listeners can adapt to these accents with exposure, it's unclear how much of this adaptation is due to general experience with accented speech versus specific experience with a particular accent. Furthermore, existing computational models of speech recognition often focus on unaccented speech and use abstract representations of mispronunciations. This paper aims to address these gaps by using a computational model (Fine-Tracker) that processes real speech and has no prior experience with foreign accents, allowing researchers to investigate the role of perceptual similarity versus experience in recognizing foreign-accented speech.

Methods

The study utilizes Fine-Tracker, a computational model of spoken-word recognition that takes actual acoustic speech signals as input. Fine-Tracker operates on two levels: a prelexical level where the acoustic signal is converted into "articulatory feature" (AF) vectors, and a lexical level where these features are matched against a lexicon of words represented in AF vectors. The model was tested in two conditions: an "inexperienced condition" simulating a listener with no prior exposure to German-accented Dutch, and an "exposure condition" simulating brief exposure. Parameter tuning for the model was performed using development sets of accented and unaccented speech. Recognition success was defined as the target word appearing at the top of the model's N-best list (N=1).

Datasets

The study used audio stimuli from a previous cross-modal priming experiment involving German-accented Dutch. This included:

  • 24 weakly-accented words
  • 12 medium-accented /ɛɪ/ words
  • 12 strongly-accented /œy/ words

For training the model's prelexical level (ANNs), 3,410 randomly selected utterances from the Spoken Dutch Corpus (CGN) were used. Development sets for parameter tuning consisted of 94 weakly-accented filler words (for the inexperienced condition) and an additional six /œy/ and six /ɛɪ/ filler words (for the exposure condition). The Dutch lexicon used contained 27,740 entries.

Key Insights and Ideas

  • Perceptual Similarity Matters: The model's performance in the "inexperienced" condition, where it showed difficulty with strongly-accented words but no significant difference between medium and weakly-accented words, suggests that perceptual similarity between the accented pronunciation and the canonical Dutch pronunciation plays a significant role in recognition ease.
  • Exposure is Key, But How? While brief exposure (simulated by parameter tuning) slightly improved the model's performance, it still struggled to match human listeners' rapid adaptation to strongly-accented words. This highlights that the way exposure is implemented in models might be crucial.
  • Model vs. Human Adaptation: The discrepancy between the model and human listeners in the exposure condition suggests that human adaptation might involve more than just parameter adjustments, potentially including prelexical adaptation to the specific accent.
  • Item-Level vs. Global Performance: Fine-Tracker could not successfully simulate listener behavior on an item-by-item basis, indicating that factors beyond phonetic detail, such as lexical frequency, might be important for fine-grained modeling.

Key Results

  • Inexperienced Condition: Fine-Tracker showed a higher average depth in its N-best list for strongly-accented words (5.2) compared to weakly-accented words (1.8), mirroring human difficulty. The percentage of target words recognized as first best was 71.4% for weak, 50% for medium, and 33.3% for strong accents.
  • Exposure Condition: Performance improved slightly, with 76.2% of weak, 58.3% of medium, and 41.7% of strong accents recognized as first best. However, the model still ranked strongly-accented words significantly lower than weakly-accented words, unlike human listeners who showed comparable recognition for all accent types after exposure.
  • Item-Level Analysis: Correlations between the model's N-best list depth and human priming effects were not significant, indicating a failure to simulate item-specific recognition patterns.

Limitations and Future Work

  • Simulation of Exposure: The authors acknowledge that implementing "exposure" through parameter tuning might not be the optimal approach. Future work will focus on incorporating exposure at the prelexical level by training Fine-Tracker on German-accented Dutch data.
  • "Monolingual" Status: Further research is needed to definitively establish Fine-Tracker's status as a true "monolingual" model of Dutch, given the complexities of real-world language exposure.
  • Item-Specific Factors: The inability to simulate item-by-item recognition suggests that other word-level factors, such as lexical frequency, need to be incorporated into future modeling efforts.
  • Human Listener Nuances: The authors suggest that the "inexperienced" condition in their simulation might more accurately reflect a Dutch speaker with truly no prior exposure to German-accented Dutch, as human listeners in the study had some limited prior experience.

Summary updated

BibTeX
@inproceedings{scharenborg2012computational,
  author = {Odette Scharenborg and Marijt Witteman and Andrea Weber},
  title = {{Computational modelling of the recognition of foreign-accented speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure