A hierarchical Bayesian approach for semi-supervised discriminative language modeling

Yik-Cheung Tam, Paul VozilaInterspeech 2012

TL;DR

This paper tackles the challenge of improving discriminative language models (DLMs) by leveraging abundant untranscribed speech data. They propose a novel semi-supervised approach using hierarchical Bayesian domain adaptation, treating manually transcribed data and automatically generated "pseudo-truth" transcriptions as distinct domains. This method allows for more effective DLM training by jointly learning from both data sources, leading to improved speech recognition performance.

What Problem Does This Paper Address?

Traditional discriminative language models (DLMs) excel at correcting speech recognition errors by learning to differentiate between correct and incorrect word hypotheses. However, DLMs typically require manually transcribed speech data for training, which is expensive and time-consuming to obtain. This creates a bottleneck for developing and deploying these powerful models. Meanwhile, vast amounts of untranscribed speech data are readily available from deployed speech recognition systems. The challenge is to effectively utilize this untranscribed data to enhance DLMs without the prohibitive cost of manual transcription.

Methods

The core of this paper is the application of Hierarchical Bayesian Domain Adaptation to semi-supervised discriminative language modeling. Here's a breakdown:

  • Discriminative Language Modeling (DLM): The paper starts by explaining DLMs, which model the conditional probability of a word sequence given acoustic features, $P(W|X)$. This is in contrast to traditional language models that model $P(W)$. DLMs use feature functions that capture acoustic scores, language model scores, and N-gram counts from N-best lists generated by a speech recognizer.
  • Feature Selection: To manage the large number of N-gram features, they employ a perceptron-based feature selection method to identify useful features.
  • Hierarchical Bayesian Domain Adaptation: This is the novel contribution. The authors treat manually transcribed data and pseudo-truth transcriptions (generated automatically from untranscribed data) as two distinct "domains." They leverage a hierarchical Bayesian framework where domain-specific models share a common prior, and this prior is also estimated from the data. This allows for a co-regularization effect, where domain models influence the prior, and the prior influences the domain models. This joint estimation is key.
  • Semi-Supervised Learning: By combining a small amount of high-quality manual transcription with a large amount of lower-quality pseudo-truth transcription, they achieve a semi-supervised learning scenario.
  • Hyper-parameter Tuning: The paper emphasizes the importance of setting hyper-parameters (which control the regularization strength between domains and the prior) to achieve different training scenarios, from independent domain training to joint adaptation.

Datasets

  • Training Data:
    • 60 hours of manual transcription.
  • 3000 hours of pseudo-truth transcription, generated using wider decoding beams and offline acoustic/language model adaptation on untranscribed speech.
  • Heldout Set: 10 hours of manual transcription, randomly sampled from the same pool as the training data.
  • Test Set: 37.8 hours of test data.

The paper uses an "in-house speech recognition task" with a "modern ASR system." The pseudo-truth transcription is generated from untranscribed speech data, differing from manual transcription by its automatic generation process.

Key Insights and Ideas

  • Pseudo-truth as a valuable proxy: The paper demonstrates that automatically generated "pseudo-truth" transcriptions, while imperfect, can be effectively leveraged to significantly improve discriminative language models, bridging the gap created by the cost of manual transcription.
  • Hierarchical Bayesian adaptation for domain synergy: The hierarchical Bayesian approach allows for a more nuanced integration of different data domains compared to simpler methods like fixed priors. The joint estimation of domain models and the prior allows them to mutually inform each other, leading to better performance.
  • Prior bias matters: Unlike methods that assume equal importance across domains (like EasyAdapt), biasing the prior model towards the non-target domain (pseudo-truth) can lead to better performance on the target domain (manual transcription). This suggests that the relevance of the non-target domain to the target task is crucial for effective adaptation.
  • Robustness to domain mismatch: Even when the pseudo-truth transcription is less aligned with the test set (as seen in Table 2), the hierarchical Bayesian approach still offers some performance gain and appears robust to such mismatches.

Key Results

  • Heldout Set Performance (Table 1):
    • Training solely on 60 hours of manual transcription yielded a 1.13% relative Word Error Rate (WERR) reduction.
  • Training solely on 3000 hours of pseudo-truth transcription yielded a 1.07% relative WERR reduction.
  • Combining manual and pseudo-truth with a fixed pseudo-truth prior (scenario c) achieved a 1.75% relative WERR reduction.
  • The proposed Hierarchical Bayesian domain adaptation, with a specific hyper-parameter setting (σ:σ₁:σ₂ = 1:0.1:0.01), achieved the best performance with a 2.12% relative WERR reduction*.
  • Test Set Performance (Table 2):
    • On the test set, pseudo-truth transcription alone showed a slight degradation (-0.8% WERR).
  • However, combining manual and pseudo-truth data using the best Hierarchical Bayesian configuration (d4) still provided a slight gain (0.78% WERR reduction), indicating robustness.

Limitations and Future Work

  • Domain Mismatch: The authors acknowledge that the performance on the test set was not as strong as on the heldout set, suggesting a potential mismatch between the pseudo-truth data and the test conditions. They explicitly state that in the future, they will "investigate the domain mismatch issue."
  • Hyper-parameter Sensitivity: While they found optimal hyper-parameter settings, the paper implies that proper tuning is crucial and can lead to different training scenarios. Further exploration of hyper-parameter optimization strategies could be beneficial.
  • Feature Sparsity: Due to data sparsity, only unigram and bigram features were extracted. Investigating the impact of higher-order N-gram features with more data or advanced feature extraction techniques could be a future direction.

Summary updated

BibTeX
@inproceedings{tam2012hierarchical,
  author = {Yik-Cheung Tam and Paul Vozila},
  title = {{A hierarchical Bayesian approach for semi-supervised discriminative language modeling}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure