Unsupervised lattice-based acoustic model adaptation for speaker-dependent conversational telephone speech transcription

K. Thambiratnam, F. SeideInterspeech 2009

TL;DR

This paper introduces a more robust method for adapting speech recognition models to individual speakers using conversational phone recordings. By leveraging "lattices" (multiple possible transcriptions) instead of just the single best guess, they achieve significant improvements in transcription accuracy, especially when dealing with noisy or imperfect data.

What Problem Does This Paper Address?

Training accurate speech recognition models for every individual speaker is ideal but often impractical due to the massive amount of labeled data required. Unsupervised adaptation, where models learn from unlabeled recordings of a specific speaker, offers a solution. However, a major challenge is that automatically generated transcripts used for adaptation can contain errors. Traditional methods that discard entire utterances or words based on confidence scores can be wasteful. This paper tackles the problem of improving unsupervised adaptation by developing a more intelligent way to use the available data, even when the automatic transcriptions are imperfect. They focus specifically on conversational telephone speech, where data collection is becoming easier but transcription accuracy remains a hurdle.

Methods

The core innovation is lattice adaptation. Instead of relying on a single, potentially erroneous transcript, this method uses a "lattice" – a data structure that represents multiple possible word sequences and their associated probabilities generated by the speech recognition system.

Here's how it works:

  • Standard Adaptation: Typically, adaptation algorithms like MLLR (Maximum Likelihood Linear Regression) and MAP (Maximum A Posteriori) use frame posteriors (the probability of being in a specific acoustic state at a given time frame) derived from a single transcript.
  • Lattice Adaptation: This paper replaces the single transcript with a lattice. Frame posteriors are then computed by considering all paths within the lattice, weighted by their probabilities. This allows the adaptation process to "smooth over" errors by distributing the probability mass across multiple hypotheses, rather than assigning it all to a single incorrect path.
  • Iterative and Cascaded Adaptation: The authors also explore performing adaptation iteratively (refining the model multiple times) and in a cascaded manner (applying different adaptation techniques sequentially).
  • Confidence Thresholding Investigation: A key part of their work is investigating whether explicitly discarding low-confidence frames or words from the lattice is beneficial. They explore various thresholding strategies (word-level, phone-level, state-level).

Datasets

  • Switchboard-1 (SWB1): Used for experiments. A subset of 62 speakers was selected.
  • Fisher English Corpus Part 1 and 2 (Fisher): Used for training baseline acoustic and language models.

For each speaker, 60 minutes of adaptation data and 10 minutes of evaluation data were used.

Key Insights and Ideas

  • Lattices are Better Than Single Transcripts: The most significant insight is that using lattices for adaptation consistently outperforms using a single, automatically generated transcript. This is because lattices provide a richer, more nuanced representation of the speech signal, allowing the adaptation process to be more robust to recognition errors.
  • Thresholding Isn't Always Necessary: Surprisingly, the paper demonstrates that explicit confidence thresholding of lattice paths is not required to achieve optimal adaptation. Accumulating statistics from the "local best-confidence path" (the most likely path within a small segment) is sufficient. This simplifies the process and avoids potentially discarding useful information.
  • Frame-Level Weighting is Key: Lattice adaptation inherently provides frame-level weighting. This means that frames corresponding to uncertain or misrecognized words contribute less to the adaptation process, which is a more principled approach than discarding entire utterances or words.
  • Iterative Adaptation Benefits: Iterative adaptation, where the model is refined over multiple passes, yields significant gains, especially for MAP adaptation. However, it's crucial to use the original unadapted model as the prior in MAP to ensure stability.
  • Cascaded Adaptation Works Well: Combining different adaptation techniques (e.g., MLLR followed by MAP) in a cascaded fashion further improves performance, with lattice-based methods showing benefits at each stage.

Key Results

  • Word Error Rate (WER) Reduction: A 3-stage cascaded lattice MLLR+MAP system reduced the WER by 7.0% absolute compared to an unadapted baseline (from 34.0% to 27.0%).
  • Gain Over Transcript Adaptation: This cascaded lattice system achieved an additional 0.8% absolute gain over the best unsupervised transcript-based adaptation system.
  • Reduced Supervised/Unsupervised Gap: The gap between supervised and unsupervised adaptation was reduced. For example, for the rtree adaptation method, the gap decreased from 2.2% to 1.5%. For MAP, it decreased from 1.6% to 0.9%.
  • Thresholding Impact: While word-based thresholding showed some minor gains in adaptation stability, it was not essential for achieving optimal accuracy. The non-thresholded lattice adaptation converged to nearly optimal results.

Limitations and Future Work

  • Computational Cost: Iterative adaptation, while beneficial, increases computational cost as it requires re-decoding after each iteration. The paper notes that for transcript adaptation, re-decoding between iterations didn't significantly impact accuracy, making it computationally cheaper. However, for lattice adaptation, re-decoding was found to create sparse lattices, which can be suboptimal.
  • Explanation for Global MLLR Gains: The paper notes an interesting observation that global MLLR systems showed gains with iterations, even though adaptation statistics are pooled. The authors state that no proper explanation for this has been determined and it remains an area for further investigation.
  • Future Work: The paper implicitly suggests further exploration into optimizing the cascaded adaptation stages and understanding the precise mechanisms behind the observed gains, particularly for global MLLR. The robustness of lattice adaptation across different acoustic conditions and languages could also be a future research direction.

Summary updated

BibTeX
@inproceedings{thambiratnam2009unsupervised,
  author = {K. Thambiratnam and F. Seide},
  title = {{Unsupervised lattice-based acoustic model adaptation for speaker-dependent conversational telephone speech transcription}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure