Semi-Supervised Discriminative Language Modeling with Out-of-Domain Text Data

Arda Çelebi, Murat SaraçlarNAACL 2013

TL;DR

This paper explores using readily available out-of-domain (OOD) text data to train discriminative language models (DLMs) for automatic speech recognition (ASR), achieving performance comparable to using limited in-domain data. They demonstrate that even randomly selected OOD data can significantly improve ASR transcription accuracy, opening up new avenues for leveraging vast amounts of unlabeled text.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems rely heavily on language models (LMs) to predict the most likely sequence of words given acoustic information. Discriminative Language Modeling (DLM) is a technique that further refines ASR by learning to distinguish between acoustically similar word sequences. However, traditional DLM requires a substantial amount of transcribed speech data and a well-trained ASR system to generate "real" hypotheses for training. This is often a bottleneck, as obtaining such high-quality, in-domain data is expensive and time-consuming. The paper addresses this by investigating whether "simulated" hypotheses, generated from out-of-domain text data, can effectively train DLMs, thereby reducing the reliance on scarce in-domain resources.

Methods

The core of the paper's methodology lies in simulating ASR hypotheses and then using these to train a DLM.

  • Hypothesis Simulation: This process involves generating alternative, potentially incorrect, transcriptions for a given reference sentence. It's a three-step pipeline:
  • Confusion Model (CM): A confusion model is first estimated from supervised data. This model captures the types of errors an ASR system typically makes (e.g., confusing "to" with "too"). This CM is then used to generate a large set of confusable word sequences.
  • Language Model Reweighting: To filter out implausible sequences generated in the first step, language models are used to reweight the hypotheses. The paper explores using the ASR system's own language model (GEN-LM), a language model trained on ASR output (ASR-LM), or no language model at all (NO-LM).
  • Sampling: From the potentially large set of generated hypotheses, a smaller, diverse subset is selected. The paper compares several sampling strategies, including Top-N (highest scoring), Uniform Sampling (US), RCxM (clustered sampling), and ASRdist-N (sampling to match the word error distribution of real ASR output).
  • DLM Estimation: The simulated hypotheses and their corresponding reference sentences are then used to train a DLM. The paper employs a WER-sensitive perceptron algorithm, which is designed to minimize word error rate directly, rather than just classification errors.
  • Out-of-Domain (OOD) Data Selection: A key contribution is the use of OOD text data. To select relevant sentences from a large corpus of OOD text, the paper uses perplexity-based filtering. Sentences with lower perplexity (when evaluated by an in-domain LM) are considered more similar to the in-domain data and are prioritized. They compare methods like TOP-N (most similar), BOTTOM-N (least similar), and RC-NxM (clustered selection).

Datasets

  • In-Domain Data: Turkish broadcast news transcription data set, split into training, held-out, and test subsets. The training set is further divided for confusion model estimation and hypothesis generation.
  • Out-of-Domain (OOD) Data: A large dataset of 10.8 million sentences (140 million morphs) from newspaper articles downloaded from the internet.

Key Insights and Ideas

  • OOD Data is a Viable Substitute: The most significant insight is that large amounts of OOD text data, even when randomly selected, can be effectively used to train DLMs, achieving performance comparable to using limited in-domain data. This significantly broadens the applicability of DLM.
  • Random Selection Can Be Sufficient: Contrary to expectations, randomly selecting OOD sentences (RAND-500K) often performed as well as, or even better than, methods that tried to select sentences most similar to the in-domain data (TOP-500K). This suggests that the sheer volume of OOD data can compensate for a lack of strict domain similarity.
  • ASRdist-50 Sampling is Robust: The ASRdist-50 sampling method, which aims to match the word error distribution of simulated hypotheses to that of real ASR output, consistently performed well, both for in-domain and OOD data.
  • Syllable-based Confusion Models Excel: Syllable-based confusion models, when paired with appropriate language models, yielded better results than morph or word-based models, suggesting that finer-grained phonetic confusion modeling is beneficial.
  • Merging Real and Simulated Hypotheses: Combining real in-domain hypotheses with simulated OOD hypotheses can lead to further improvements, though the gains might be marginal compared to using solely simulated OOD data.

Key Results

  • Comparable WER Reduction: The study found that using OOD data for hypothesis simulation achieved WER reductions comparable to those obtained with in-domain data. For example, RAND-500K (OOD) achieved similar results to the in-domain simulated hypotheses (t2).
  • ASRdist-50 Outperforms Other Sampling Methods: ASRdist-50 achieved the lowest average rank for sampling methods, indicating superior WER reduction on OOD data.
  • Syllable/Morph CMs Outperform Word CMs: Syllable and morph-based confusion models had a similar average rank (1.5), significantly better than the word-based model (2.8).
  • OOD Data Size Matters: As the size of the OOD data set increased (from 50K to 500K), the WER reduction improved and eventually surpassed that of in-domain data.
  • Merging Benefits: Merging real hypotheses from t1 with simulated hypotheses from RAND500K achieved a WER of 22.0% on the held-out set, comparable to using simulated hypotheses from t2.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while OOD data can be effective, the "best" OOD data selection method is still an open question.
  • The performance gains from merging real and simulated hypotheses were sometimes marginal, especially on the test set.
  • The syllable-based CM paired with NO-LM showed outlier behavior, suggesting that the interaction between confusion model granularity and language model usage is complex.
  • Future Work:
    • The authors suggest increasing the size of the OOD data further.
  • Exploring other OOD selection methods, such as relative entropy-based approaches, is a promising direction.
  • Investigating the interaction between different confusion model granularities and language models more deeply.

Summary updated

BibTeX
@inproceedings{celebi2013semi,
  author = {Arda Çelebi and Murat Saraçlar},
  title = {{Semi-Supervised Discriminative Language Modeling with Out-of-Domain Text Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure