Modeling cue trading in human word recognition

Louis ten Bosch, Odette ScharenborgInterspeech 2012

TL;DR

This paper introduces a computational model that quantifies "cue trading" – how different acoustic cues can be swapped to maintain speech perception – within the context of word recognition, not just phoneme identification. By adapting feature weights in an ASR-like framework, they demonstrate that this model can improve word recognition accuracy, suggesting that cue trading is a crucial mechanism for human speech processing.

What Problem Does This Paper Address?

Human speech perception is remarkably robust, allowing us to understand speech even when acoustic signals are noisy or distorted. A key phenomenon contributing to this robustness is "cue trading," where different acoustic features (like the way a sound is produced or its frequency characteristics) can compensate for each other. For example, a slight change in vowel duration might be compensated by a change in tongue position to ensure the listener perceives the same phoneme. While cue trading has been extensively studied for individual phoneme identification, its role in the more complex task of recognizing entire words has been less understood. This paper aims to bridge this gap by developing a computational model that can quantify cue trading effects during word recognition and investigate how these trading relationships are learned and utilized.

Methods

The authors propose a computational model inspired by the Fine-Tracker model, which integrates a prelexical and a lexical level.

  • Prelexical Level: This level processes the incoming acoustic signal and converts it into streams of articulatory features (AFs). This is achieved using an ensemble of artificial neural networks (ANNs), each trained to estimate the presence or absence of specific AFs over time.
  • Lexical Level: This level uses a Hidden Markov Model (HMM) recognition system, common in Automatic Speech Recognition (ASR). Crucially, the model represents words as sequences of "lexical AF vectors," where some features can be "underspecified" (not explicitly defined).
  • Cue Trading Mechanism: The core of the model's ability to handle cue trading lies in how it calculates the "goodness of fit" (GOF) between prelexical AFs and lexical AFs. This GOF is influenced by feature weights, which are derived from the variances of the HMM's Gaussian distributions. Initially, specified features have low variance (high weight), and unspecified features have high variance (low weight).
  • Automatic Adaptation: The model leverages an ASR training paradigm (Expectation-Maximization) to automatically adjust these variances (and thus feature weights) based on real speech data. This adaptation process allows the model to learn which features are most important for distinguishing words, effectively quantifying cue trading.

Datasets

  • Training Data: 5386 utterances from the Dutch part of the ACORNS database were used for adjusting the Gaussian variances (training the model). This data was manually transcribed and labeled at the phoneme level.
  • Test Data: 600 utterances from the same ACORNS database were used for testing the model and investigating cue trading.
  • Language Model Data: A bigram language model was built using 4490 utterances from a separate set with similar syntactical structure.

Key Insights and Ideas

  • Quantifying Cue Trading: The model provides a novel way to quantitatively measure cue trading by observing how feature weights (inversely related to variances) change during training. This allows for a direct assessment of which features are traded and to what extent.
  • Word Recognition is Key: The study successfully demonstrates that cue trading is not just relevant for phoneme identification but plays a significant role in the more complex task of word recognition.
  • Automatic Learning of Feature Importance: The ASR training process automatically learns which articulatory features are most discriminative for word recognition. This highlights the adaptive nature of human speech perception.
  • Asynchrony Matters: The paper suggests that the temporal dynamics and asynchrony of features are important for understanding cue trading, with some features showing early or late activation patterns.

Key Results

  • Improved Recognition Accuracy: The model's word recognition performance, measured by Word Error Rate (WER), significantly improved after training. The WER decreased from 21.33% (baseline) to 12.25% after 5 iterations of training. This improvement directly supports the idea that cue trading is occurring and beneficial.
  • Dominance of Manner and Place Features: The analysis of feature weights revealed that "manner" and "place" of articulation features gained the most significant weights during training, indicating they are the most crucial for distinguishing phonemes within this word recognition task.
  • Feature Undershoot: The analysis of feature dynamics showed that actual AF values at the prelexical stage often "undershoot" the target canonical lexical AF values, further supporting the idea of compensatory trading.

Limitations and Future Work

  • Phoneme-Specific Cue Trading: The current model provides phoneme-dependent cue weightings, but the authors plan to further investigate how cue trading specifically differs across various phonemes.
  • Data Sparseness: To refine the analysis of feature weighting and asynchrony, more speech data might be needed to avoid issues related to data sparseness.
  • Refining GOF: The authors plan to apply the learned cue weightings to refine the goodness-of-fit mechanism in the Fine-Tracker model, potentially leading to further improvements.
  • Broader Applications: The authors suggest that ASR-based speech analysis methods inspired by human speech processing can be valuable for analyzing large speech corpora and bridging the gap between computational models of human speech and ASR.

Summary updated

BibTeX
@inproceedings{bosch2012modeling,
  author = {Louis ten Bosch and Odette Scharenborg},
  title = {{Modeling cue trading in human word recognition}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure