A Multistage Training Framework for Acoustic-to-Word Model

Chengzhu Yu, Chunlei Zhang, Chao Weng, Jia Cui, Dong Yu

Abstract

Acoustic-to-Word (A2W) models simplify automatic speech recognition (ASR) by mapping audio directly to words. However, these models often struggle with data sparsity and performance on moderate-sized datasets. This paper introduces a multistage training framework that incorporates hierarchical pretraining, curriculum learning for vocabulary expansion, and a joint CTC-CE loss. On the Switchboard 300-hour task, this framework achieves an 11.4% Word Error Rate (WER), representing a 12.1% relative improvement over baseline A2W models, without the use of an external language model or complex decoder.

Introduction

Traditional ASR systems are complex pipelines involving separate acoustic, lexicon, and language models. Acoustic-to-Word (A2W) models aim to simplify this by mapping audio directly to words in a single end-to-end neural network.

However, A2W models face two major hurdles:

  1. Data Sparsity: Many words appear rarely in training data, making them hard to learn.
  2. Scale: While tech giants have shown A2W works with 100,000+ hours of data, these models often underperform on moderate datasets (like the standard 300-hour Switchboard corpus) compared to traditional systems.

The authors tackle how to make A2W models competitive when data is limited by providing better "hints" to the model during training.

Methodology

The authors use a CLDNN architecture (CNNs for feature extraction, followed by Bidirectional LSTMs and a linear projection layer). Their multistage framework consists of three main pillars:

Hierarchical CTC Pretraining

Instead of training on words from scratch, the model is initialized in stages. First, the bottom layers are trained to recognize phonemes. Then, additional layers are added and trained to recognize graphemes. Finally, the whole stack is used to initialize the word-level model. This forces the network to learn the underlying phonetic and symbolic structure of language before tackling whole words.

Hierarchical initialization process from phones to graphemes to words.

Curriculum Training

The model starts by learning only the 10,000 most frequent words. Utterances containing rarer words are initially discarded. Once the model converges, the vocabulary is expanded to 20,000 words. This "easy-to-hard" approach ensures a stable convergence point and facilitates better learning of the expanded vocabulary.

Joint CTC-CE Training

The model is trained using two loss functions simultaneously: Connectionist Temporal Classification (CTC) and Cross-Entropy (CE). They propose a specific architecture where the final output is a concatenation of two separate linear projection layers, allowing the model to benefit from the strengths of both objectives.

Comparison of Vanilla CTC, Multi-task learning, and the proposed Joint CTC-CE network.

Experimental Setup

The framework was evaluated using the 300-hour Switchboard English speech corpus. Final models utilized 3-fold data augmentation via speed perturbation. Performance was measured on the Hub5-2000 test sets, specifically the Switchboard (SWB) and CallHome (CH) subsets.

Results

The combination of these techniques resulted in a final A2W system that is highly competitive:

  • Baseline A2W: 14.8% WER (SWB) / 25.8% WER (CH)
  • Final Multistage Model: 11.4% WER (SWB) / 20.8% WER (CH)
  • Relative Improvement: A 12.1% relative reduction in WER over the baseline.

Notably, their A2W model outperformed other end-to-end approaches like "Attention Seq2seq" (25.8% WER) and "Iterated CTC" (15.1% WER) on the same 300-hour task.

Discussion and Conclusion

Key insights from the study include:

  • Structure Matters: Simply adding more layers to a word-level model doesn't help much. However, using those layers to learn intermediate representations (phones and graphemes) provides a massive performance boost.
  • Vocabulary "Warm-up": Increasing the vocabulary size from 10k to 20k without curriculum training yielded almost no gain. With curriculum training, it led to a significant 1% absolute reduction in deletion errors.
  • The Power of Simplicity: The final model achieves impressive results without an external Language Model (LM).

While the results are a major step forward, a gap still exists between this A2W approach and the very best conventional systems (which reach ~9.6% WER). Future work will involve scaling this multistage framework to larger datasets, such as the 2,000-hour Fisher corpus.

Summary updated

Abstract

Acoustic-to-word (A2W) prediction model based on Connectionist Temporal Classification (CTC) criterion has gained increasing interest in recent studies. Although previous studies have shown that A2W system could achieve competitive Word Error Rate (WER), there is still performance gap compared with the conventional speech recognition system when the amount of training data is not exceptionally large. In this study, we empirically investigate advanced model initializations and training strategies to achieve competitive speech recognition performance on 300 hour subset of the Switchboard task (SWB-300Hr). We first investigate the use of hierarchical CTC pretraining for improved model initialization. We also explore curriculum training strategy to gradually increase the target vocabulary size from 10k to 20k. Finally, joint CTC and Cross Entropy (CE) training techniques are studied to further improve the performance of A2W system. The combination of hierarchical-CTC model initialization, curriculum training and joint CTC-CE training translates to a relative of 12.1% reduction in WER. Our final A2W system evaluated on Hub5-2000 test sets achieves a WER of 11.4/20.8 for Switchboard and CallHome parts without using language model and decoder.
BibTeX
@inproceedings{yu2018multistage,
  author = {Chengzhu Yu and Chunlei Zhang and Chao Weng and Jia Cui and Dong Yu},
  title = {{A Multistage Training Framework for Acoustic-to-Word Model}},
  booktitle = {Interspeech},
  year = {2018},
  doi = {10.21437/interspeech.2018-1452},
}
Figure