Abstract
Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional sub-word based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.
Introduction
Traditional Automatic Speech Recognition (ASR) systems are complex, typically requiring separate acoustic, pronunciation, and language models. Even modern "end-to-end" systems often rely on sub-word units (like characters or phonemes), necessitating a complex decoding process and external language models to achieve high accuracy. Direct Acoustics-to-Word (A2W) models offer a simpler alternative by mapping audio directly to word units. However, due to the sparsity of word-level labels compared to phonemes, these models have historically required massive datasets (often 100,000+ hours) to match the performance of hybrid systems. On standard 2,000-hour benchmarks like Switchboard, A2W models have previously lagged behind. This work introduces a training "recipe" and a novel architecture to close this performance gap.
The Optimized Training Recipe
The researchers identified a combination of training techniques that significantly improve A2W performance. Key components include:
- Curriculum Learning: Sorting training data by ascending length (shortest utterances first) proved critical for helping the network learn simpler patterns before tackling complex sequences.
- Initialization: Acoustic layers (BLSTMs) are initialized from a pre-trained phone-based model, and the final word-output layer is initialized using GloVe word embeddings to provide semantic grounding.
- Architecture Tweaks: The model uses a 6-layer Bidirectional LSTM with an Output Projection Layer. Since the vocabulary is large (25,000+ words), decomposing the final linear layer into two smaller ones speeds up training and reduces overfitting.
The Spell and Recognize (SAR) Model
To address the challenge of Out-of-Vocabulary (OOV) words, the authors proposed the Spell and Recognize (SAR) model. Unlike standard A2W models that might simply fail or omit unknown words, the SAR model is trained to output a sequence consisting of the word's spelling followed by the word itself (e.g., [spell] c a t [word] CAT). This joint modeling ensures that even if a word is not in the fixed vocabulary, the model can provide a phonetically recognizable character-based hypothesis.
Results and Discussion
Using the optimized recipe on the 2,000-hour Switchboard-Fisher dataset, the A2W model achieved a Word Error Rate (WER) of 8.8% on Switchboard and 13.9% on CallHome. This represents a significant improvement over previous benchmarks (13.0% / 18.8%) and places the model at par with state-of-the-art hybrid models that use complex decoders and external language models.
The SAR model demonstrated its utility by providing readable hypotheses for OOV words. For instance, when encountering the unseen word "SCHOLARLY," the model produced the spelling "COLARLY" before an <UNK> tag, offering a useful approximation for the user rather than a blank space.
Conclusion
This work demonstrates that with proper initialization, curriculum learning, and architectural refinements, direct A2W models can be highly competitive with traditional sub-word systems even on moderately sized datasets. The SAR model further enhances these systems by providing a mechanism to handle rare and unseen words through joint spelling and recognition. Future work will focus on improving spelling accuracy and exploring data-driven methods for character-to-word mapping.
Summary updated