Acoustic-to-Phrase Models for Speech Recognition

Yashesh Gaur, Jinyu Li, Zhong Meng, Yifan Gong

Introduction

In the evolution of Automatic Speech Recognition (ASR), the units predicted by models—known as "tokens"—have progressively increased in scale. Systems have transitioned from phonetic sounds to characters, sub-words (word-pieces), and eventually whole words. This paper investigates the feasibility of extending this progression to multi-word phrases, exploring whether end-to-end (E2E) models can transcribe directly into acoustic-to-phrase units.

Modeling frequent phrases (e.g., "on the way" or "hey Cortana") as single tokens offers two primary advantages. First, it allows the model to capture how words are co-articulated as a single acoustic unit. Second, it significantly enhances inference speed for Sequence-to-Sequence (Seq2Seq) models, which generate text one token at a time. By representing multiple words with a single token, the number of decoding steps is reduced, lowering latency.

Methodology

The researchers compared two dominant E2E architectures: Connectionist Temporal Classification (CTC) and attention-based Sequence-to-Sequence (Seq2Seq) models. While CTC assumes conditional independence between output tokens, the Seq2Seq decoder functions as an integrated language model, tracking dependencies between predictions.

Three phrase-based vocabularies—P2, P3, and P4—were constructed, containing phrases up to two, three, and four words long, respectively. A "longest-string-first" tokenization strategy was employed: if a multi-word phrase from the vocabulary was present in the transcript, the model was trained to emit that single phrase token rather than individual word tokens.

Comparison of tokenization strategies across different vocabulary types, illustrating how a single utterance is decomposed into various token units.

The models were trained on a massive industrial dataset comprising approximately 3,400 hours of Microsoft Cortana voice assistant tasks (3.3 million short utterances) in US English. Performance was evaluated against baselines using characters, words, word-pieces, and mixed-units.

Results and Discussion

The study revealed a stark contrast between the two architectures. Seq2Seq models proved remarkably robust to larger token units, whereas CTC performance degraded significantly as phrase length increased.

The Impact of the Decoder

The success of Seq2Seq models is attributed to the decoder's ability to model dependencies. CTC’s "conditional independence" assumption makes it ill-suited for tokens that vary widely in acoustic duration. As phrases grew longer, the CTC Word Error Rate (WER) skyrocketed from approximately 9.84% for words to 24.87% for the P4 vocabulary. In contrast, the Seq2Seq model's WER only increased slightly, from 7.51% (mixed-units) to 8.21% (P4).

Word Error Rate (WER) and model size comparison for S2S and CTC architectures across various token types, highlighting the failure of CTC on phrase-level tokens.

Efficiency and Token Distribution

By utilizing phrases, the output sequence length is drastically reduced. Analysis of token emission showed that Seq2Seq models effectively utilized the multi-word tokens in proportions similar to the training data. For instance, in the P4 model, 7.5% of emitted tokens were 4-grams. CTC models, however, struggled to emit these complex units effectively, further explaining their poor performance.

Conclusion

This research demonstrates that Seq2Seq models can effectively target multi-word phrases with minimal impact on accuracy, potentially enabling much faster decoding. While the current "longest-first" tokenization strategy is effective, future work may involve optimizing vocabularies to better balance accuracy and speed. Additionally, empirical wall-clock speed benchmarks are needed to fully quantify the latency benefits of phrase-level emission in production environments.

Summary updated

Abstract

Directly emitting words and sub-words from speech spectrogram has been shown to produce good results using end-to-end (E2E) trained models. Connectionist Temporal Classification (CTC) and Sequence-to-Sequence attention (Seq2Seq) models have both shown better success when directly targeting words or sub-words. In this work, we ask the question: Can an E2E model go beyond words and transcribe directly to phrases (i.e., a group of words)? Directly modeling frequent phrases might be better than modeling its constituent words. Also, emitting multiple words together might speed up inference in models like Seq2Seq where decoding is inherently sequential. To answer this, we undertake a study on a 3400-hour Microsoft Cortana voice assistant task. We present a side-by-side comparison for CTC and Seq2Seq models that have been trained to target a variety of tokens including letters, sub-words, words and phrases. We show that an E2E model can indeed transcribe directly to phrases. We see that while CTC has difficulty in accurately modeling phrases, a more powerful model like Seq2Seq can effortlessly target phrases that are up to 4 words long, with only a reasonable degradation in the final word error rate.
BibTeX
@inproceedings{gaur2019acoustic,
  author = {Yashesh Gaur and Jinyu Li and Zhong Meng and Yifan Gong},
  title = {{Acoustic-to-Phrase Models for Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3056},
}
Figure