Abstract
Statistical language model, trained by a large number of text corpus, is an integral component in many speech and natural language model processing systems, such as speech recognition and machine translation. It is a probabilistic model which describes the distribution pattern of natural language. Over the last few decades, N-gram language model (LM) is the most popular technique since it is simple and effective. However, the training of the N-gram language model is based on the maximum likelihood rule resulting in suboptimal output in speech recognition systems. In this paper, a discriminative training based language model (DLM) which directly focused on minimizing speech recognition word error rate (WER) was employed to improve the performance of speech recognition system. In particular, the part-of-speech (POS) feature was used to train DLM as well as the n-gram features. Experimental results showed that DLM with n-gram features gave 1% absolute reduction in word error rate (WER). Combining n-gram features with POS feature, DLM could obtain another 0.4% absolute reduction in WER.
Introduction
Most modern speech recognition systems rely on N-gram language models trained using Maximum Likelihood Estimation (MLE). While effective, MLE has a fundamental flaw: it focuses on making the correct text likely but does not explicitly learn to distinguish the correct transcription from "near-miss" errors. This often leads to suboptimal outputs where the model's top choice is grammatically or contextually inferior to other candidates in its "n-best" list.
Mandarin Chinese presents additional challenges due to its complex structure and the lack of clear word boundaries in raw text. This research aims to bridge this gap by using a discriminative approach that specifically targets the minimization of the Word Error Rate (WER) rather than just maximizing likelihood.
Methodology
The proposed system utilizes a two-pass recognition framework designed to refine initial hypotheses using discriminative features.
- The First Pass: A standard Large Vocabulary Continuous Speech Recognition (LVCSR) system uses a trigram language model to generate a "word lattice," which is then pruned into an n-best list (the top 1,000 most likely hypotheses).
- The Second Pass (Re-scoring): A Discriminative Language Model (DLM) is used to re-evaluate these 1,000 candidates. The DLM calculates a new score for each hypothesis based on the original score from the first pass, N-gram features (counts of specific unigrams, bigrams, and trigrams), and POS features (syntactic information extracted using the ICTCLAS lexical analysis tool).
The model is trained using the Perceptron algorithm. This incremental algorithm predicts the best hypothesis for a training example; if the prediction is incorrect (i.e., not the one with the lowest WER), it adjusts feature weights to penalize the error and reward the correct transcription.
Experimental Setup
The study utilized a massive amount of data to ensure robustness:
- Acoustic/Language Modeling: 13 million words of transcription data and 2 billion words of generic web data were used for the initial N-gram models.
- DLM Training: 34,018 utterances.
- Development/Optimization: 3,200 utterances of Mandarin Chinese spontaneous conversational speech.
- Evaluation: 2,476 utterances of spontaneous Mandarin speech.
Key Results and Discussion
The core advantage of discriminative training is that the model learns from both correct and incorrect examples. By analyzing erroneous candidate transcriptions, the model identifies and penalizes common acoustic or linguistic confusions. Furthermore, adding Part-of-Speech (POS) tags provides syntactic smoothing, allowing the model to make better decisions based on grammatical structure even when specific word sequences are novel.
The researchers evaluated the system on a 1000-best list re-scoring task. The results showed a clear stepwise improvement as shown in the experimental data.
| Model Approach | WER (%) | Absolute Reduction |
|---|---|---|
| Baseline (Trigram) | 49.6% | - |
| DLM (N-grams only) | 48.6% | 1.0% |
| DLM (N-grams + POS) | 48.2% | 1.4% |
While a 1.4% absolute reduction might seem modest, in the context of LVCSR, these gains are significant and often represent the difference between a usable and an unusable system in noisy or spontaneous conversational environments.
Conclusion
This paper demonstrates that incorporating POS features into a Discriminative Language Model effectively improves Mandarin speech recognition. By moving beyond standard "most likely" word sequences to a re-ranking approach that minimizes WER, the system achieves better performance. Future work will explore more advanced features like semantic relevance and compare the Perceptron algorithm against other discriminative training methods to further reduce the high baseline WER associated with spontaneous conversational Mandarin.
Summary updated