Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Michael Picheny

TL;DR

Direct acoustics-to-word (A2W) models simplify speech recognition by mapping audio directly to word units, but they typically require massive datasets to be effective. This paper introduces a refined training "recipe" and a novel "Spell and Recognize" (SAR) architecture that allows these models to achieve state-of-the-art performance on the 2,000-hour Switchboard dataset—matching complex hybrid systems without needing a separate language model or decoder.

What Problem Does This Paper Address?

Traditional Automatic Speech Recognition (ASR) is a "multi-headed beast" consisting of an acoustic model (audio to phonemes), a pronunciation model (phonemes to words), and a language model (word sequences). Even many "end-to-end" models still rely on sub-word units like characters, which requires a complex decoding process and an external language model to reach high accuracy.

Direct Acoustics-to-Word (A2W) models are the "holy grail" of simplicity: audio goes in, words come out. However, because words are much sparser than phonemes, these models usually require orders of magnitude more data (often 100,000+ hours) to perform well. On standard benchmarks like the 2,000-hour Switchboard set, A2W models have historically lagged 3–4% behind traditional systems. This paper aims to close that gap.

Methods

The authors attack the problem from two angles: a rigorous training optimization "recipe" and a new model architecture.

1. The Optimized Training Recipe

The authors identified several "tricks" that, when combined, significantly boost performance:

  • Initialization: Instead of starting from scratch, they initialize the acoustic layers (BLSTMs) using a pre-trained phone-based model and initialize the final word-output layer using GloVe word embeddings.
  • Curriculum Learning: They found that sorting training data by ascending length (shortest utterances first) was critical. This allows the network to learn simpler patterns before tackling long, complex sentences.
  • Architecture Tweaks: They used a 6-layer Bidirectional LSTM and added an Output Projection Layer. Since the vocabulary is huge (25,000+ words), decomposing the final linear layer into two smaller ones speeds up training and reduces overfitting.

2. The Spell and Recognize (SAR) Model

To handle Out-of-Vocabulary (OOV) words—words the model didn't see during training—they proposed the SAR model. Instead of just predicting "CAT," the model is trained to output a sequence: [spell] c a t [word] CAT. By forcing the model to spell the word before identifying it, the system can still provide a readable character-based hypothesis even if the specific word isn't in its vocabulary (emitting an <UNK> tag).

Datasets

  • Switchboard-1 (300 hours): Used for rapid experimentation and ablation studies.
  • Fisher + Switchboard + CallHome (2,000 hours): The primary dataset for state-of-the-art comparison.
  • Hub5-2000: The standard evaluation set, consisting of Switchboard (SWB) and CallHome (CH) subsets.

Key Insights and Ideas

  • Data Order Matters: Changing the training sequence from descending to ascending length resulted in the single largest performance gain. This suggests that "warming up" on short sequences helps the model find a better local optimum in the parameter space.
  • The Power of Initialization: A2W models struggle because they try to learn acoustics and semantics simultaneously. Initializing with phone-based models provides the necessary "acoustic grounding" to make word-level learning feasible on smaller datasets.
  • Joint Modeling vs. Multi-Tasking: Unlike Multi-Task Learning (MTL), where words and characters are predicted on separate heads (often leading to synchronization issues), the SAR model's interleaved sequence ensures that the spelling and the word are perfectly aligned in time.

Key Results

The updated recipe allowed the A2W model to achieve a Word Error Rate (WER) of 8.8% on Switchboard and 13.9% on CallHome.

  • This is a massive improvement over their previous best of 13.0% / 18.8%.
  • Crucially, these results are at par with state-of-the-art hybrid models that use complex decoders and external language models, even though the A2W model uses "greedy" decoding (simply picking the most likely word at each step).
  • The SAR model successfully handled OOV words. For example, when it encountered the word "SCHOLARLY" (unseen), it spelled out "COLARLY" before the <UNK> tag, providing the user with a phonetically recognizable result rather than a blank space.

Limitations and Future Work

  • Spelling Accuracy: While the SAR model provides a "best guess" for OOV words, the spelling isn't always perfect (e.g., "COLARLY" instead of "SCHOLARLY").
  • Fixed Vocabulary: The "word" portion of the model is still limited by a fixed vocabulary (25,000 words in this study).
  • Future Directions: The authors suggest using data-driven methods to further refine the character-to-word mapping and improve the accuracy of the "spelling" phase for rare words.

Summary updated

Abstract

Direct acoustics-to-word (A2W) models in the end-to-end paradigm have received increasing attention compared to conventional subword based automatic speech recognition models using phones, characters, or context-dependent hidden Markov model states. This is because A2W models recognize words from speech without any decoder, pronunciation lexicon, or externally-trained language model, making training and decoding with such models simple. Prior work has shown that A2W models require orders of magnitude more training data in order to perform comparably to conventional models. Our work also showed this accuracy gap when using the English Switchboard-Fisher data set. This paper describes a recipe to train an A2W model that closes this gap and is at-par with state-of-the-art sub-word based models. We achieve a word error rate of 8.8.8%/13.9% on the Hub5-2000 Switchboard/CallHome test sets without any decoder or language model. We find that model initialization, training data order, and regularization have the most impact on the A2W model performance. Next, we present a joint word-character A2W model that learns to first spell the word and then recognize it. This model provides a rich output to the user instead of simple word hypotheses, making it especially useful in the case of words unseen or rarely-seen during training.
BibTeX
@inproceedings{audhkhasi2018building,
  author = {Kartik Audhkhasi and Brian Kingsbury and Bhuvana Ramabhadran and George Saon and Michael Picheny},
  title = {{Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=OodREleb6l},
}

Loading related content...

Figure