Introduction
In the evolution of Automatic Speech Recognition (ASR), the units predicted by models—known as "tokens"—have progressively increased in scale. Systems have transitioned from phonetic sounds to characters, sub-words (word-pieces), and eventually whole words. This paper investigates the feasibility of extending this progression to multi-word phrases, exploring whether end-to-end (E2E) models can transcribe directly into acoustic-to-phrase units.
Modeling frequent phrases (e.g., "on the way" or "hey Cortana") as single tokens offers two primary advantages. First, it allows the model to capture how words are co-articulated as a single acoustic unit. Second, it significantly enhances inference speed for Sequence-to-Sequence (Seq2Seq) models, which generate text one token at a time. By representing multiple words with a single token, the number of decoding steps is reduced, lowering latency.
Methodology
The researchers compared two dominant E2E architectures: Connectionist Temporal Classification (CTC) and attention-based Sequence-to-Sequence (Seq2Seq) models. While CTC assumes conditional independence between output tokens, the Seq2Seq decoder functions as an integrated language model, tracking dependencies between predictions.
Three phrase-based vocabularies—P2, P3, and P4—were constructed, containing phrases up to two, three, and four words long, respectively. A "longest-string-first" tokenization strategy was employed: if a multi-word phrase from the vocabulary was present in the transcript, the model was trained to emit that single phrase token rather than individual word tokens.
The models were trained on a massive industrial dataset comprising approximately 3,400 hours of Microsoft Cortana voice assistant tasks (3.3 million short utterances) in US English. Performance was evaluated against baselines using characters, words, word-pieces, and mixed-units.
Results and Discussion
The study revealed a stark contrast between the two architectures. Seq2Seq models proved remarkably robust to larger token units, whereas CTC performance degraded significantly as phrase length increased.
The Impact of the Decoder
The success of Seq2Seq models is attributed to the decoder's ability to model dependencies. CTC’s "conditional independence" assumption makes it ill-suited for tokens that vary widely in acoustic duration. As phrases grew longer, the CTC Word Error Rate (WER) skyrocketed from approximately 9.84% for words to 24.87% for the P4 vocabulary. In contrast, the Seq2Seq model's WER only increased slightly, from 7.51% (mixed-units) to 8.21% (P4).
Efficiency and Token Distribution
By utilizing phrases, the output sequence length is drastically reduced. Analysis of token emission showed that Seq2Seq models effectively utilized the multi-word tokens in proportions similar to the training data. For instance, in the P4 model, 7.5% of emitted tokens were 4-grams. CTC models, however, struggled to emit these complex units effectively, further explaining their poor performance.
Conclusion
This research demonstrates that Seq2Seq models can effectively target multi-word phrases with minimal impact on accuracy, potentially enabling much faster decoding. While the current "longest-first" tokenization strategy is effective, future work may involve optimizing vocabularies to better balance accuracy and speed. Additionally, empirical wall-clock speed benchmarks are needed to fully quantify the latency benefits of phrase-level emission in production environments.
Summary updated