Performance comparison of training algorithms for semi-supervised discriminative language modeling

Erinç Dikici, Arda Çelebi, Murat SaraçlarInterspeech 2012

TL;DR

This paper explores semi-supervised training for Discriminative Language Models (DLMs) by comparing six perceptron-based algorithms. They found that ranking-based perceptrons generally outperform structured perceptrons, especially when trained with real data, and that combining real and simulated data yields competitive results.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems rely on language models (LMs) to predict the most likely sequence of words. Discriminative Language Models (DLMs) aim to improve ASR accuracy by directly learning to distinguish between acoustically similar word sequences. However, training DLMs typically requires a large amount of transcribed speech data and a well-performing ASR system, which are often difficult to obtain. This paper addresses this challenge by investigating semi-supervised training methods, which leverage simulated hypotheses instead of real transcribed data to train DLMs.

Methods

The core of this research lies in comparing different training algorithms for DLMs within a semi-supervised framework. The authors formulate DLM as both a structured prediction and a reranking problem.

  • Semi-Supervised Hypothesis Generation: Instead of using real ASR outputs, they generate simulated hypotheses. This process involves:
  • Confusion Modeling: Creating a model that captures how acoustically similar language units (like phonemes or words) are confused.
  • Language Model Reweighting: Using a language model to reweight the generated hypotheses, favoring more linguistically plausible sequences.
  • Hypothesis Sampling: Selecting a subset of hypotheses from a larger generated set to reduce computational cost without significantly impacting accuracy.
  • Training Algorithms: They compare six perceptron variants:
  • Structured Perceptrons: Algorithms designed to pick the best hypothesis from a set (e.g., Per, WPer, RPer).
  • Ranking Perceptrons: Algorithms that rerank the entire set of hypotheses, aiming to push more accurate ones to the top (e.g., PerRank, WPerRank, RPerRank). These algorithms optimize a pairwise comparison between hypotheses.

Datasets

The study uses a Turkish ASR system and its associated dataset, divided as follows:

  • Training: Approximately 188 hours of ASR output. For semi-supervised training, this was further split into $t_1$ (for confusion model construction) and $t_2$ (for generating simulated hypotheses).
  • Held-out (Validation): 3.1 hours of ASR output.
  • Test: 3.3 hours of ASR output.

The paper focuses on using simulated N-best lists generated from the $t_2$ portion of the training data for semi-supervised experiments.

Key Insights and Ideas

  • Ranking vs. Structured: Ranking perceptron algorithms generally achieve better or comparable performance to structured perceptrons, especially when trained with real data. This suggests that reranking the entire hypothesis list is a more effective approach for DLM than simply picking the best one.
  • Data Combination is Key: While simulated data alone doesn't fully match the performance of real data, combining real and simulated data for training yields competitive results. This highlights the potential of semi-supervised methods to alleviate the need for extensive real transcribed data.
  • Algorithm Nuances: Different variants of the perceptron (e.g., WER-sensitive vs. reciprocal) show varying degrees of effectiveness, indicating that the specific update rules and loss functions matter.
  • Training Time Consideration: Structured perceptrons might be preferred over ranking perceptrons when training time is a critical constraint, as they can be computationally less demanding.

Key Results

  • Real Data Performance: When trained on real ASR 50-best lists, WPerRank achieved the lowest Word Error Rate (WER) on the held-out set, especially when the baseline score was included in training.
  • Semi-Supervised Performance: In semi-supervised experiments, ranking perceptrons (RPerRank) generally achieved better average ranks than structured perceptrons. While the WER reduction was not always significant compared to structured perceptrons, they showed a tendency towards better minimum WERs.
  • Data Combination: Combining real and simulated data for training resulted in competitive test set WERs, often approaching the performance of models trained solely on real data. For instance, "Real Sim" training achieved a test WER of 21.6%, very close to "Real Real" at 21.5%.
  • Structured Perceptron Weakness: The structured perceptron (Per) was found to be the worst performer across most semi-supervised scenarios.

Limitations and Future Work

  • Limitations:
    • The superiority of ranking algorithms was less pronounced when using simulated data compared to real data.
  • The structured perceptron (Per) consistently performed poorly in semi-supervised settings.
  • Future Work: The authors suggest that further investigation is needed to understand the WE distributions of training tests and their impact on model accuracy. They also implicitly point towards the benefit of combining real and simulated data as a promising direction.

Summary updated

BibTeX
@inproceedings{dikici2012performance,
  author = {Erinç Dikici and Arda Çelebi and Murat Saraçlar},
  title = {{Performance comparison of training algorithms for semi-supervised discriminative language modeling}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure