A frame level boosting training scheme for acoustic modeling

Rong Zhang, Alexander I. RudnickyInterspeech 2004 · OpenReview

Abstract

Conventional Boosting algorithms for acoustic modeling have two notable weaknesses. (1) The objective function aims to minimize utterance error rate, though the goal for most speech recognition systems is to reduce word error rate. (2) During Boosting training, an utterance is treated as a unit for re-sampling and each frame within the same utterance is assigned equal weight. Intuitively, the frames associated with a misclassified word should be given more emphasis than others. We propose a frame level Boosting training scheme that addresses these shortcomings and allows each frame to have a different weight. We describe a technique and provide experimental results for this approach.

Introduction

In the early 2000s, researchers began applying Boosting—an ensemble technique that combines multiple "weak" learners into a strong one—to acoustic modeling. However, standard boosting algorithms for speech had two major flaws. First, they typically minimized the Utterance Error Rate (UER). In speech recognition, a hypothesis is only "correct" if every single word is right, which is an incredibly strict metric. Most systems prioritize Word Error Rate (WER), but standard boosting doesn't distinguish between an utterance with one wrong word and one where every word is wrong. Second, in utterance-level boosting, if a sentence is wrong, every frame (time slice) in that sentence gets an equal weight increase. Intuitively, the training process should only boost the specific frames associated with the misclassified words.

Proposed Frame-Level Boosting

The proposed Frame Level Boosting scheme provides finer granularity by shifting the focus from entire utterances to individual frames.

Frame-Level Probabilities and Objective Function

Instead of looking at the probability of a whole transcript $P(h|x)$, the system calculates the conditional probability of a specific word being correct at a specific frame $t$, denoted as $P(a_t|x)$. This is estimated using N-best lists generated by the decoder, making frame-level probability estimation computationally feasible. A designed loss function minimizes frame-based recognition error; if a frame is "confusing," it receives a higher weight in the next round of training.

Resampling and Alignment

To train subsequent models in the ensemble, data is resampled at the frame level by duplicating frames with high weights. This forces the new acoustic model to learn difficult segments more effectively. Since ground truth word-to-frame correspondence is not naturally available, forced alignment is used to generate labels for every frame in the training set.

Experimental Setup

The study used the CMU Communicator corpus, a collection of telephone-based dialogs regarding travel planning. The training set consisted of 31,248 utterances, while the test set included 1,689 utterances from a NIST evaluation. The experiments were conducted using the Carnegie Mellon Sphinx-3 decoder, a continuous HMM system.

Results and Discussion

The performance of the frame-level scheme was compared against a single baseline model and standard utterance-level boosting across different ensemble sizes ($K$).

Word Error Rate (WER) results for Utterance-Level Boosting as the number of models (K) increases.
Word Error Rate (WER) results for the proposed Frame-Level Boosting scheme across different ensemble sizes (K).

As shown in the results, both boosting methods significantly outperformed the single baseline model (14.99% WER). Most of the gains were achieved with an ensemble of 3 to 4 models, after which returns diminished. While the frame-level scheme showed substantial improvement over the baseline, reaching 13.52% WER at $K=4$, it performed slightly worse than the simpler utterance-level boosting (13.27% WER).

Conclusion

The core insight of this work is that speech is non-stationary; some parts of an utterance are easy to recognize (like silence), while others are difficult (like noisy consonants). Boosting at the frame level allows the trainer to focus on these "hard" parts. The slightly lower performance compared to utterance-level boosting may be due to imperfect labels from forced alignment or the crude nature of frame duplication for resampling. Future work suggests a hybrid approach that considers both utterance-level and frame-level errors simultaneously to capture both global context and local signal difficulties.

Summary updated

Abstract

Conventional Boosting algorithms for acoustic modeling have two notable weaknesses. (1) The objective function aims to minimize utterance error rate, though the goal for most speech recognition systems is to reduce word error rate. (2) During Boosting training, an utterance is treated as a unit for re-sampling and each frame within the same utterance is assigned equal weight. Intuitively, the frames associated with a misclassified word should be given more emphasis than others. We propose a frame level Boosting training scheme that addresses these shortcomings and allows each frame to have a different weight. We describe a technique and provide experimental results for this approach.
BibTeX
@inproceedings{zhang2004frame,
  author = {Rong Zhang and Alexander I. Rudnicky},
  title = {{A frame level boosting training scheme for acoustic modeling}},
  booktitle = {Interspeech},
  year = {2004},
  url = {https://openreview.net/forum?id=Gyzd2c4136},
}
Figure