Exploring discriminative speech trajectory structures

Heyun Huang, Louis ten Bosch, Bert Cranen, Lou BovesInterspeech 2012

TL;DR

This paper explores how to improve speech sound (phone) recognition by explicitly modeling the continuous, gradual movement of the vocal tract. They introduce a "smoothness regularizer" to supervised dimensionality reduction techniques, finding it helps distinguish vowels but hinders consonant recognition, suggesting different acoustic properties require different modeling approaches.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle with accurately distinguishing between similar speech sounds (phones). While the physical production of speech involves continuous, relatively slow movements of the articulators (tongue, lips, etc.), this inherent continuity constraint is often overlooked in traditional ASR models. Existing methods that try to incorporate trajectory information often fail to balance the risk of overfitting to the training data with the risk of creating representations that don't reflect the underlying physical process. This paper aims to address this by developing a method that explicitly enforces continuity in speech representations to improve phone discrimination.

Methods

The core of this paper is the introduction of a smoothness regularizer applied to supervised dimensionality reduction (SDR) techniques.

  • Supervised Dimensionality Reduction (SDR): The authors start with a standard SDR framework. They represent speech segments as sequences of Mel-Frequency Cepstral Coefficients (MFCCs). These multi-frame representations are then vectorized and projected into a lower-dimensional space using a projection matrix $W$. The goal is to find a $W$ that maximizes the separation between different phone classes. This is typically done by optimizing the ratio of between-class scatter to within-class scatter (as in Linear Discriminant Analysis - LDA).
  • Smoothness Regularizer: To enforce continuity, they introduce a penalty term that measures the "roughness" of the MFCC trajectories over time. This penalty is based on the second-order difference of the MFCC coefficients across frames. By minimizing this penalty alongside the standard SDR objective, they aim to obtain a low-dimensional representation that is both discriminative and physically plausible. This is mathematically formulated by adding a term proportional to $W^T D D^T W$ to the denominator of the SDR objective, where $D$ is a Laplacian smoothing matrix.
  • Compared Methods: They compare their smoothed versions of LDA (SLDA) and Locally Discriminant Embedding (LDE) (SLDE) against their non-smoothed counterparts (LDA and LDE). These methods are then used with a weighted k-Nearest Neighbor (WkNN) classifier.

Datasets

  • TIMIT: The experiments are conducted using the TIMIT corpus, a widely used dataset for acoustic-phonetic research. They use the standard NIST training set, the core test set, and the development set for parameter tuning. The original 64 phone labels are merged into 48 phone labels, excluding glottal stops.

Key Insights and Ideas

  • Vowels Benefit, Consonants Suffer: The most striking finding is that enforcing trajectory smoothness significantly improves the discrimination of vowels but degrades the performance for consonants.
  • Continuity is Key for Vowels: Vowels are generally produced with more continuous vocal tract movements. The smoothness regularizer appears to capture this inherent property, leading to better separation.
  • Discontinuity is Key for Consonants: Consonants, particularly plosives and nasals, are often characterized by abrupt changes or "event-like" transitions (e.g., the sudden release of a plosive, the opening/closing of the velum for nasals). The smoothness penalty, by forcing these trajectories to be "smoother," might obscure the very information that makes these sounds distinct.
  • Manifold Dimension: The analysis suggests that consonants intrinsically occupy a lower-dimensional manifold than vowels, meaning their acoustic variations can be described by fewer independent parameters.

Key Results

  • Vowel Improvement: For "less difficult" vowel pairs (accuracy < 0.95), SLDA and SLDE achieved accuracies of 91.41% and 91.76%, respectively, compared to 89.46% and 89.49% for LDA and LDE. Similar improvements were seen for "most difficult" pairs.
  • Consonant Degradation: For "less difficult" consonant pairs, SLDA and SLDE achieved 88.66% and 88.58%, slightly lower than LDA and LDE (89.05% and 88.66%). The degradation was more pronounced for "most difficult" consonant pairs.
  • Broad Class Performance: Table 2 provides a detailed breakdown by phonetic class, confirming that vowels (Semi-Vowels, Short Vowels, Long Vowels) generally see improvements with SLDE, while plosives and nasals show a performance drop.
  • Parameter Sensitivity: Figures 1-4 illustrate how classification accuracy varies with the reduced dimension ($r$) and the smoothness parameter ($\alpha$). For vowels, higher $\alpha$ values (more smoothing) generally lead to better performance, especially for long vowels. For consonants, the impact of $\alpha$ is less clear and often negative.

Limitations and Future Work

  • Consonant Modeling: The paper acknowledges that the current smoothness regularizer is counter-productive for certain consonants due to their event-like nature.
  • Future Work: The authors propose exploring more sophisticated trajectory models that can capture broad-class-specific or even phone-specific dynamics. They also suggest investigating the robustness of the Neumann discretizer approach to additive noise in MFCCs.

Summary updated

BibTeX
@inproceedings{huang2012exploring,
  author = {Heyun Huang and Louis ten Bosch and Bert Cranen and Lou Boves},
  title = {{Exploring discriminative speech trajectory structures}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure