Phoneme-Based Contextualization for Cross-Lingual Speech Recognition in End-to-End Models

Ke Hu, Antoine Bruguier, Tara N. Sainath, Rohit Prabhavalkar, Golan Pundak

Abstract

Contextual automatic speech recognition (ASR), which biases recognition toward specific contexts like user contacts or playlists, remains a significant challenge for end-to-end (E2E) models. These models often struggle with rare named entities, particularly foreign proper nouns that are out-of-vocabulary (OOV) relative to the training data. While standard grapheme or wordpiece models fail to spell these unseen words correctly, phonemes offer a more acoustically salient representation. This work proposes a hybrid E2E model incorporating both English wordpieces and phonemes. By mapping foreign pronunciations into a shared English phoneme space, we perform contextual biasing at the phoneme level. Experimental results demonstrate that this approach achieves a 16% relative improvement over grapheme-only biasing and an 8% improvement over wordpiece-only biasing on foreign place name recognition, with minimal impact on general English tasks.

Introduction

Modern E2E ASR models, such as RNN-Transducers, typically map acoustic signals directly to sub-word units like wordpieces. While efficient for high-frequency vocabulary, these models often fail on rare named entities—specifically foreign geographic locations (e.g., "Créteil")—because they lack the internal spelling logic for words virtually unseen during training.

Contextual biasing aims to address this by injecting a list of expected words into the decoding process. However, when the model has never encountered the orthography of these foreign words, traditional grapheme-based biasing is ineffective. This research explores using phonemes as an acoustic bridge to enable cross-lingual recognition within an English-centric model.

Methodology

The proposed system utilizes a three-pronged strategy to bridge the gap between English training data and foreign-word recognition:

  1. Hybrid Wordpiece-Phoneme Model: The model is trained to output either standard wordpieces or sequences of phonemes. During training, rare words are randomly decomposed into phoneme sequences, teaching the model to "sound out" unfamiliar acoustic patterns.
  2. Cross-Lingual Phoneme Mapping: Foreign names are converted into their native phonemes (e.g., French) and then mapped to the nearest sounding English phonemes. This allows an English-trained model to recognize the acoustic signature of a foreign word.
  3. Phoneme-Level Contextual Biasing: A Weighted Finite State Transducer (WFST) is used during decoding to bias the model toward specific phoneme sequences. These sequences are then mapped back to the correct foreign orthography for the final output.

Experimental Evaluation

The model was trained on 35 million English utterances (approximately 27,500 hours) of anonymized traffic data. Evaluation was conducted on two primary sets:

  • Directions Set: 1,000 navigation queries containing French place names, synthesized using Parallel WaveNet to simulate rare OOV entities.
  • General Traffic: 30,500 English utterances to monitor for regressions in standard performance.

Results and Discussion

The experimental results, summarized in Table 1, highlight the effectiveness of phoneme-based biasing. The phoneme-only biasing approach achieved a Word Error Rate (WER) of 27.5% on the Directions set, representing a 16% relative improvement over the grapheme baseline (32.8%) and an 8% improvement over the wordpiece baseline (30.0%). Combining wordpiece and phoneme biasing further reduced the WER to 26.9%.

Table 1: Performance comparison (WER %) of different biasing units across datasets.

A key finding is that phonemes are more acoustically salient than graphemes for OOV words. By focusing on how a word sounds rather than how it is spelled, the model becomes more robust to foreign entities. Furthermore, the model maintained high accuracy on general English tasks, with only a negligible 0.1% absolute WER increase.

However, the system's performance is sensitive to the size of the biasing list. As shown in Figure 1, the WER increases significantly as the number of biasing words grows from 1 to 1,000. This is primarily due to phonetic confusion between similar-sounding foreign names within the larger list.

Figure 1: Impact of bias list size on Word Error Rate (WER).

Conclusion

This work demonstrates that a hybrid wordpiece-phoneme model can significantly improve the recognition of foreign named entities in English ASR. By leveraging phonemes as a universal acoustic representation and employing cross-lingual mapping, the model successfully recognizes OOV words that traditional grapheme models cannot. Future work will investigate the use of longer phonemic units to reduce phonetic confusion in large-scale biasing scenarios.

Summary updated

Abstract

Contextual automatic speech recognition, i.e., biasing recognition towards a given context (e.g. user’s playlists, or contacts), is challenging in end-to-end (E2E) models. Such models maintain a limited number of candidates during beam-search decoding, and have been found to recognize rare named entities poorly. The problem is exacerbated when biasing towards proper nouns in foreign languages, e.g., geographic location names, which are virtually unseen in training and are thus out-of-vocabulary (OOV). While grapheme or wordpiece E2E models might have a difficult time spelling OOV words, phonemes are more acoustically salient and past work has shown that E2E phoneme models can better predict such words. In this work, we propose an E2E model containing both English wordpieces and phonemes in the modeling space, and perform contextual biasing of foreign words at the phoneme level by mapping pronunciations of foreign words into similar English phonemes. In experimental evaluations, we find that the proposed approach performs 16% better than a grapheme-only biasing model, and 8% better than a wordpiece-only biasing model on a foreign place name recognition task, with only slight degradation on regular English tasks.
BibTeX
@inproceedings{hu2019phoneme,
  author = {Ke Hu and Antoine Bruguier and Tara N. Sainath and Rohit Prabhavalkar and Golan Pundak},
  title = {{Phoneme-Based Contextualization for Cross-Lingual Speech Recognition in End-to-End Models}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=jXuVYuK2AZL},
}
Figure