Abstract
Contextual automatic speech recognition, i.e., biasing recognition towards a given context (e.g. user's playlists, or contacts), is challenging in end-to-end (E2E) models. Such models maintain a limited number of candidates during beam-search decoding, and have been found to recognize rare named entities poorly. The problem is exacerbated when biasing towards proper nouns in foreign languages, e.g., geographic location names, which are virtually unseen in training and are thus out-of-vocabulary (OOV). While grapheme or wordpiece E2E models might have a difficult time spelling OOV words, phonemes are more acoustically salient and past work has shown that E2E phoneme models can better predict such words. In this work, we propose an E2E model containing both English wordpieces and phonemes in the modeling space, and perform contextual biasing of foreign words at the phoneme level by mapping pronunciations of foreign words into similar English phonemes. In experimental evaluations, we find that the proposed approach performs 16% better than a grapheme-only biasing model, and 8% better than a wordpiece-only biasing model on a foreign place name recognition task, with only slight degradation on regular English tasks.
Introduction
End-to-end (E2E) automatic speech recognition (ASR) models have revolutionized the field by mapping audio directly to text within a single neural network. However, these models frequently struggle with the "rare word" problem. Because they rely on a fixed vocabulary of characters or wordpieces learned during training, they often fail to recognize out-of-vocabulary (OOV) words.
This challenge is particularly acute in contextual biasing, where the system must recognize specific entities relevant to a user's current context, such as names in a contact list or local landmarks. When these entities are foreign (e.g., an English speaker requesting directions to "Créteil" in France), standard models often fail because the spelling-to-sound relationship is unfamiliar. This research demonstrates that biasing based on sound (phonemes) rather than spelling (graphemes) is significantly more effective for these cross-lingual scenarios.
Methodology
To bridge the gap between English-trained models and foreign-language inputs, three primary innovations were implemented:
- Hybrid Wordpiece-Phoneme Model: The model's output layer is expanded to include both wordpieces and phonemes. During training, a sampling strategy is employed where common words remain as wordpieces, while rare words are randomly converted into phoneme sequences. This trains the model to output phonetic sounds when encountering unfamiliar acoustic patterns.
- Cross-Lingual Phoneme Mapping: Foreign words are handled via a two-step process. For a French word, the system retrieves its native pronunciation and then maps those sounds to the nearest English phonemes already present in the model's vocabulary.
- Phoneme-Level Contextual Biasing: At inference, the system constructs a Finite State Transducer (FST) using these mapped English phonemes. This search graph nudges the model to prefer specific phonetic sequences when the audio matches, which are subsequently converted back into the correct foreign spelling for the final output.
Experimental Setup
The model was trained on 35 million English utterances (approximately 27,500 hours) of anonymized general traffic. Performance was evaluated using two primary test sets:
- Directions Test Set: 1,000 navigation queries containing French place names (e.g., "directions to Ardeche") synthesized via a Text-to-Speech (TTS) engine.
- General Traffic Test Set: 30,500 English utterances to ensure the model maintains high performance on standard language tasks.
Results and Analysis
The proposed wordpiece-phoneme model demonstrated significant improvements over baseline grapheme and wordpiece-only models on the French directions task.
As shown in Table 1, the hybrid model achieved a 16% relative improvement in Word Error Rate (WER) compared to grapheme-only biasing and an 8% relative improvement over wordpiece-only biasing. Crucially, this was achieved with only a negligible 0.1% absolute increase in error on general English tasks, proving the robustness of the hybrid approach.
However, the effectiveness of contextual biasing is sensitive to the size of the bias list. As the number of candidate words increases, the potential for phonetic "collisions"—where different names share similar sound profiles—leads to higher error rates.
Discussion and Limitations
The results suggest that phonemes are more "acoustically salient" than letters, allowing the model to recognize sounds it can identify even if it has never seen the specific foreign spelling. By combining wordpieces and phonemes, the system captures the efficiency of wordpieces for common language while utilizing the flexibility of phonemes for the "long tail" of rare words.
Despite these gains, the system remains dependent on a pronunciation lexicon; if a word is missing from the lexicon, the system cannot currently generate a phonetic hypothesis. Future work will explore "phoneme pieces"—units larger than a single sound but smaller than a word—to reduce confusion and improve matching sparsity when dealing with very large context lists.
Summary updated