Abstract
Acoustic-to-Word (A2W) models simplify automatic speech recognition (ASR) by mapping audio directly to words. However, these models often struggle with data sparsity and performance on moderate-sized datasets. This paper introduces a multistage training framework that incorporates hierarchical pretraining, curriculum learning for vocabulary expansion, and a joint CTC-CE loss. On the Switchboard 300-hour task, this framework achieves an 11.4% Word Error Rate (WER), representing a 12.1% relative improvement over baseline A2W models, without the use of an external language model or complex decoder.
Introduction
Traditional ASR systems are complex pipelines involving separate acoustic, lexicon, and language models. Acoustic-to-Word (A2W) models aim to simplify this by mapping audio directly to words in a single end-to-end neural network.
However, A2W models face two major hurdles:
- Data Sparsity: Many words appear rarely in training data, making them hard to learn.
- Scale: While tech giants have shown A2W works with 100,000+ hours of data, these models often underperform on moderate datasets (like the standard 300-hour Switchboard corpus) compared to traditional systems.
The authors tackle how to make A2W models competitive when data is limited by providing better "hints" to the model during training.
Methodology
The authors use a CLDNN architecture (CNNs for feature extraction, followed by Bidirectional LSTMs and a linear projection layer). Their multistage framework consists of three main pillars:
Hierarchical CTC Pretraining
Instead of training on words from scratch, the model is initialized in stages. First, the bottom layers are trained to recognize phonemes. Then, additional layers are added and trained to recognize graphemes. Finally, the whole stack is used to initialize the word-level model. This forces the network to learn the underlying phonetic and symbolic structure of language before tackling whole words.
Curriculum Training
The model starts by learning only the 10,000 most frequent words. Utterances containing rarer words are initially discarded. Once the model converges, the vocabulary is expanded to 20,000 words. This "easy-to-hard" approach ensures a stable convergence point and facilitates better learning of the expanded vocabulary.
Joint CTC-CE Training
The model is trained using two loss functions simultaneously: Connectionist Temporal Classification (CTC) and Cross-Entropy (CE). They propose a specific architecture where the final output is a concatenation of two separate linear projection layers, allowing the model to benefit from the strengths of both objectives.
Experimental Setup
The framework was evaluated using the 300-hour Switchboard English speech corpus. Final models utilized 3-fold data augmentation via speed perturbation. Performance was measured on the Hub5-2000 test sets, specifically the Switchboard (SWB) and CallHome (CH) subsets.
Results
The combination of these techniques resulted in a final A2W system that is highly competitive:
- Baseline A2W: 14.8% WER (SWB) / 25.8% WER (CH)
- Final Multistage Model: 11.4% WER (SWB) / 20.8% WER (CH)
- Relative Improvement: A 12.1% relative reduction in WER over the baseline.
Notably, their A2W model outperformed other end-to-end approaches like "Attention Seq2seq" (25.8% WER) and "Iterated CTC" (15.1% WER) on the same 300-hour task.
Discussion and Conclusion
Key insights from the study include:
- Structure Matters: Simply adding more layers to a word-level model doesn't help much. However, using those layers to learn intermediate representations (phones and graphemes) provides a massive performance boost.
- Vocabulary "Warm-up": Increasing the vocabulary size from 10k to 20k without curriculum training yielded almost no gain. With curriculum training, it led to a significant 1% absolute reduction in deletion errors.
- The Power of Simplicity: The final model achieves impressive results without an external Language Model (LM).
While the results are a major step forward, a gap still exists between this A2W approach and the very best conventional systems (which reach ~9.6% WER). Future work will involve scaling this multistage framework to larger datasets, such as the 2,000-hour Fisher corpus.
Summary updated