On speaker adaptive training of artificial neural networks

Jan Trmal, Jan Zelinka, Luděk MüllerInterspeech 2010

TL;DR

This paper explores Speaker Adaptive Training (SAT) for Multilayer Perceptron (MLP) Artificial Neural Networks (ANNs) in phoneme recognition, a technique traditionally used with GMM-HMM systems. They introduce two SAT methods, VTLN and a novel Minimum Error Linear Transform (MELT), demonstrating significant improvements in phoneme error rates, especially when reference transcripts are available.

What Problem Does This Paper Address?

Speaker variability is a major challenge in automatic speech recognition (ASR), leading to a performance gap between speaker-independent (SI) and speaker-dependent (SD) systems. While Speaker Adaptive Training (SAT) is a well-established technique for improving GMM-HMM systems, its application to MLP ANNs for phoneme recognition has been largely unexplored. The authors aim to bridge this gap, investigating whether ANNs can inherently learn speaker normalization and to what extent this ability is influenced by network size and the temporal context of the input features (TRAPS).

Methods

The paper presents two primary methods for speaker adaptation within an MLP ANN framework:

  • Vocal Tract Length Normalization (VTLN): This standard technique adjusts the frequency bins of the mel-filter bank to compensate for variations in vocal tract length. The authors apply VTLN factors during the TRAPS feature construction phase.
  • Minimum Error Linear Transform (MELT): This novel approach treats speaker normalization as a linear transformation applied directly to the ANN's input weight matrix. It aims to find a transformation matrix that minimizes an error criterion, analogous to how ANNs are typically trained. To combat potential overtraining due to the large number of parameters in the MELT matrix, they propose a dimensionality reduction technique by expressing the transform as a function of a smaller vector.

Both methods are used to adapt a pre-trained SI ANN, followed by retraining the ANN with the adapted features.

Datasets

  • SpeechDat-East: This telephone speech corpus, containing recordings from approximately 1000 speakers, was used for training and evaluation.
  • Training: Approximately 900 speakers were used for training the SI ANN and for the SAT experiments.
  • Testing: 200 speakers were used for testing.
  • The phonetic alphabet consisted of 37 three-state phonemes.
  • A zerogram language model was used to eliminate language model influence.

Key Insights and Ideas

  • ANNs can learn speaker normalization: The experiments suggest that MLP ANNs, to some extent, can learn to compensate for speaker variability without explicit external normalization techniques.
  • Network size matters, but has limits: While larger ANNs (2500 neurons vs. 1500) show better performance, the improvement plateaus, indicating that network size alone doesn't fully address speaker normalization.
  • TRAPS features are not sufficient for full normalization: The fact that SAT-trained ANNs outperform SI ANNs suggests that a single TRAPS frame (spanning ~300ms) doesn't contain all the necessary information for complete speaker normalization.
  • MELT outperforms VTLN: The novel MELT approach consistently achieved better results than VTLN, implying that speaker variability is not solely characterized by vocal tract length and that a more direct feature-level transformation can be more effective.

Key Results

The paper reports significant improvements in phoneme error rate (PER) reduction:

  • With reference transcript (supervised normalization):
    • SAT-VTLN achieved up to 25% relative error rate reduction.
  • SAT-MELT achieved up to 25% relative error rate reduction.
  • Without reference transcript (unsupervised normalization):
    • SAT-VTLN achieved up to 12% relative error rate reduction.
  • SAT-MELT achieved up to 12% relative error rate reduction.

The results are presented in terms of accuracy (ACC), with tables showing performance after each SAT iteration, indicating convergence. For instance, for the ANN-2500, SAT-VTLN improved accuracy from 79.25% (base) to 82.07% (after 4 iterations), while SAT-MELT improved it to 83.60%.

Limitations and Future Work

  • Overtraining with MELT: The direct application of the MELT algorithm is prone to overtraining, especially with limited speaker-specific data. The proposed dimensionality reduction is a step towards mitigating this, but it's acknowledged that there's no universally simple way to design the relationship between the reduced vector and the transformation matrix.
  • Unsupervised normalization potential: While unsupervised normalization showed improvements, the authors believe more sophisticated unsupervised approaches could yield even better results.
  • Focus on phoneme recognition: The experiments were specifically targeted at phoneme recognition within a hybrid MLP-HMM framework. The authors suggest that using ANN-derived bottleneck features with standard GMM-SAT systems could be a promising avenue for future research.
  • Speaker variability beyond vocal tract length: The superiority of MELT over VTLN suggests that speaker variability encompasses more than just vocal tract length, and further research into characterizing and modeling these other aspects is warranted.

Summary updated

BibTeX
@inproceedings{trmal2010speaker,
  author = {Jan Trmal and Jan Zelinka and Luděk Müller},
  title = {{On speaker adaptive training of artificial neural networks}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure