TL;DR
This paper introduces a novel method for training discriminative language models (DLMs) without requiring any transcribed speech data. By leveraging "phrasal cohorts" derived from untranscribed speech, they achieve significant improvements in speech recognition accuracy, recovering over 40% of the gains seen in fully supervised methods. This opens doors for more accessible and scalable DLM training.
What Problem Does This Paper Address?
Automatic speech recognition (ASR) systems heavily rely on language models (LMs) to predict the likelihood of word sequences. While traditional generative LMs (like n-grams) are easy to train on large text corpora, they don't directly optimize for ASR performance metrics like Word Error Rate (WER). Discriminative LMs (DLMs) aim to directly minimize WER by learning from potential recognition errors.
However, training DLMs typically requires transcribed speech to generate "confusion sets" – lists of likely erroneous outputs for a given correct utterance. Acquiring large amounts of transcribed speech is expensive and time-consuming, limiting the widespread adoption of DLMs, especially in new domains or low-resource scenarios. This paper tackles the challenge of training effective DLMs in a fully unsupervised setting, where no transcribed speech is available.
Methods
The core innovation is the phrasal cohort method for generating confusion sets without transcribed data.
- Phrasal Cohorts: These are sets of similar-sounding phrases that could be confused with each other.
- Supervised Extraction (from [1]): In the presence of transcribed speech, phrasal cohorts are extracted by aligning the reference transcript with the ASR system's top hypotheses. Differences between the reference and hypothesis, bounded by matching "pivot" words, define the transformation rules (e.g., "that to you" confused with "that too you").
- Unsupervised Extraction (Novelty): For the fully unsupervised case, the authors extend this by extracting cohorts from pairs of hypotheses within an N-best list generated by the ASR system on untranscribed speech. Since the correct utterance is unknown, the transformation rules are extracted symmetrically (e.g., if "what kind" is confused with "well kind," both "what kind -> well kind" and "well kind -> what kind" rules are learned).
- Hallucination of N-best Lists: Once phrasal cohorts are extracted (either supervised or unsupervised), they are used to "hallucinate" confusion sets. This involves constructing a confusion network and using a language model to extract the top N hypotheses for a given text.
- Local N-best Training (Efficiency): To speed up experiments and hyperparameter tuning, a "local N-best" training strategy is used. Instead of building a full confusion network, for each training sentence, potential confusions are generated by replacing phrases with their cohort counterparts. This is computationally less intensive than global N-best list generation.
- Discriminative Training: The hallucinated confusion sets are then used to train a discriminative language model, typically using an averaged perceptron algorithm with unigram, bigram, and trigram features.
Datasets
- English Conversational Telephone Speech (CTS) dataset: Used for experiments.
- NIST RT04 Fall development and evaluation sets: Used for reporting results.
- Baseline ASR system: Acoustic models trained on 2000 hours of transcribed data.
- Baseline LM: 4-gram LM trained on 2000 hours of transcribed speech and 1 billion words of web text.
- Training Data:
- Supervised DLM: Uses 690 hours of transcribed speech.
- Unsupervised DLM (Transcribed Cohorts): Uses 690 hours of transcribed speech to derive cohorts.
- Unsupervised DLM (Untranscribed Cohorts): Uses untranscribed speech (derived from the same 690 hours but without reference transcripts) to derive cohorts.
- LM Text: Up to 10 million words of text used for hallucinating confusion sets.
Key Insights and Ideas
- The power of phrasal cohorts extends to unsupervised learning: The most significant finding is that the phrasal cohort method, originally proposed for semi-supervised settings, can be effectively adapted for fully unsupervised DLM training. This is a major step towards making DLMs more accessible.
- Unsupervised DLMs can complement, but not significantly boost, supervised DLMs: When some transcribed data is available, training an unsupervised DLM on top of a supervised DLM derived from that same data does not yield further improvements and can sometimes even degrade performance. This suggests that if transcribed data is available, it's best to focus on maximizing the supervised DLM.
- Symmetric cohort extraction is a viable, albeit coarser, unsupervised approach: While less precise than supervised extraction, deriving cohorts symmetrically from untranscribed speech still allows for meaningful confusion set generation and leads to performance gains.
- Local N-best training is efficient for tuning: The local N-best strategy proved useful for hyperparameter tuning without the computational burden of global N-best list generation.
Key Results
- Unsupervised DLMs with transcribed cohorts: Achieved 0.6% absolute WER reduction, recovering over half of the gains of the supervised approach.
- Unsupervised DLMs with untranscribed cohorts (Fully Unsupervised): Consistently achieved approximately 0.4% absolute WER reduction across various data conditions. This represents over 40% of the gains of the supervised method, demonstrating the effectiveness of the proposed approach in the absence of any transcribed data.
- Low-resource performance: Even with as little as 17 hours of speech, the unsupervised DLM with untranscribed cohorts achieved 0.5% WER improvement, outperforming the standard supervised DLM.
- Data scaling: The benefits of unsupervised training seemed to saturate quickly with the amount of data, suggesting that the method is efficient even with limited resources.
Limitations and Future Work
- Scalability of phrasal cohorts: The symmetric definition of phrasal cohorts in the unsupervised setting can lead to very large sets. For future work, the authors plan to explore pruning strategies to make large-scale experiments (e.g., 1000 hours of speech, 100 million words of text) more feasible.
- Symmetric extraction noise: The authors note that the symmetric extraction of cohorts from untranscribed data can introduce noise, which might explain why the global N-best extraction (using an LM) performed better than the local training strategy in the unsupervised case. Further investigation into mitigating this noise could be beneficial.
Summary updated