Monolingual Pretraining Bias in Speech Foundation Models

26 concepts25 papers1 proposal

Self-supervised learning (SSL) has fundamentally transformed speech and audio processing. Models like HuBERT and WavLM learn powerful, general-purpose acoustic representations by predicting masked segments of continuous audio waveforms. However, because the massive datasets used to train these foundation models are predominantly English-centric, the models develop an implicit structural bias. They do not merely learn the universal physics of sound; instead, their latent spaces over-index on the specific phonemes, prosodic structures, and syntactic patterns of a single language.

When these models are deployed globally, this phenomenon creates a severe cross-lingual generalization gap. The learned acoustic features fail to transfer effectively to typologically distinct languages, creating a systemic bottleneck across speech recognition, compression, paralinguistics, and generative audio frameworks.

The Anatomy of the Representation Bottleneck

To understand why monolingual pretraining limits a model's utility, we must look at how speech is discretized and encoded in latent space.

When an SSL model is trained on a single language, its internal clustering mechanisms optimize for the acoustic boundaries that matter in that language. For example, an English-centric model will fine-tune its representations to distinguish between English phonemes but may completely collapse the acoustic variations required to distinguish tones in Mandarin or click consonants in Xhosa.

Continuous Audio ──> [ Monolingual SSL Encoder ] ──> Latent Space Warped Toward English Phonetics
                                                              │
                                                              └──> Fails to resolve non-English 
                                                                   phonemic & tonal boundaries

This structural blind spot manifests clearly when trying to extract discrete speech tokens—the audio equivalent of words in natural language processing. As Yifan Yang et al. demonstrate in Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS, discrete tokens derived from monolingual self-supervised models fail to generalize to different linguistic structures or phoneme sets. The discrete codebook simply lacks the vocabulary to represent non-English phonetic structures cleanly.

This limitation is a core challenge for speech language models. Xin Zhang et al. show in SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models that tokenizers trained predominantly on a single language face severe challenges capturing the phonetic and prosodic nuances of diverse linguistic families.

The representation gap propagates through various specialized acoustic tasks:

As Hagai Aronowitz et al. observe in Towards a Common Speech Analysis Engine, this single-language pretraining restricts immediate global applicability, forcing practitioners into resource-intensive downstream fine-tuning to adapt acoustic representations to different linguistic structures. Even when models scale up to massive datasets, if the data remains English-centric, the underlying encoder's inherent generalization capabilities remain fundamentally limited (Cheol Jun Cho et al.).

Downstream Cascades: Codecs, Paralinguistics, and Generation

The downstream consequences of monolingual pretraining bias are widespread, impacting neural audio codecs, paralinguistic analysis, and generative speech models.

Neural Audio Codecs and Compression

Modern neural audio codecs compress audio by mapping continuous waveforms to a low-bitrate discrete space, frequently using a semantic teacher model to guide the reconstruction of meaningful content. However, if the teacher model is blind to non-English phonetic structures, the codec's optimization objective is fundamentally flawed for global deployment.

Rongkun Xue et al. demonstrate in HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling that the effectiveness of neural audio codecs is strictly constrained by the linguistic scope of the semantic models used for distillation, which are frequently biased toward English.

As a result, when a codec uses a self-supervised representation reconstruction loss to preserve high-level semantic features, the performance faces a strict ceiling in multilingual contexts if the teacher model cannot capture universal phonetic structures (Junhyeok Lee et al.). This results in a failure to generalize to the phonetic and tonal nuances of diverse global languages, a limitation that Jun Xu et al. argue must be solved by expanding evaluations to broader linguistic variations in Rate-Aware Learned Speech Compression.

Paralinguistics and Emotion Recognition

Paralinguistic tasks like Speech Emotion Recognition (SER) rely heavily on prosody, pitch, and subtle vocal inflections—features that vary significantly across cultures and languages. Payal Mohapatra et al. show in Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Tasks that English-only pretraining creates a severe performance bottleneck when models must capture emotional nuances across diverse global speech patterns.

This lack of cross-lingual robustness is a recurring theme. Encoders trained primarily on a single language lack the cross-lingual robustness found in massive multilingual models, restricting their effectiveness in global applications (Aharon Satt et al.). When moving to generative architectures, Soham Deshmukh et al. show in SELM: Enhancing Speech Emotion Recognition for Out-of-Domain Scenarios that generative emotion models trained exclusively on a single language suffer significant performance degradation when applied to multilingual contexts, exposing a major gap in cross-lingual audio-to-text emotion mapping.

Generative Speech and Audio LLMs

The same bottleneck restricts generative speech foundation models and Large Audio Language Models (LALMs). Frameworks for task composition (Justin Lovelace et al.) and high-quality speech restoration (Pin-Jui Ku et al.) face strict limitations in generalizing to global speech patterns due to the lack of phonetic and acoustic diversity in their monolingual pretraining data.

Furthermore, LALMs optimized primarily for English fail to scale to global contexts (Sreyan Ghosh et al.). This issue is compounded by a historical lack of non-English evaluation resources, a gap that Hui Li et al. address in MDAR: A Multi-scene Dynamic Audio Reasoning Benchmark by developing benchmarks specifically to test the generalizability of audio language models in non-English scenarios.

Even multimodal frameworks suffer from this foundational bias. Andrew Rouditchenko et al. observe in mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition that the heavy English focus in audio-visual speech recognition leads to significant performance degradation when deployed in multilingual or non-English contexts, particularly in noisy environments. Similarly, when building contextual biasing systems, prompt-based filtering mechanisms frequently lack validation across non-English phonetic structures (Guanrou Yang et al.).

Overcoming the Bias

To close the cross-lingual generalization gap, researchers are exploring several architectural and data-driven interventions that move beyond simple fine-tuning.

Monolingual Pretraining ──> Latent Space Bias ──> Downstream Performance Collapse
                                 ▲
Mitigation Strategies:           │
  1. External Language Embeddings ┼───────────────────────────────┐
  2. Synchronous Bilingual CTC    ┼───────────────────────────────┤
  3. Multilingual Pretraining     ┼─ [Sanyuan Chen et al. (WavLM)] ┘

One approach is to explicitly inject external linguistic knowledge during the pretraining or transfer phase. Junrui Ni et al. demonstrate that incorporating a language's phylogenetic relations (its "family tree") and its phoneme inventory can mitigate the bias introduced by a single dominant language backbone, significantly improving zero-shot transfer to unrelated languages.

Alternatively, models can leverage multi-task cross-lingual regularization. In Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognition, Chen Xu et al. demonstrate that forcing an encoder to simultaneously predict transcripts and translations allows the model to utilize the semantic richness of the target language to regularize and improve its understanding of the original audio.

Ultimately, moving past the monolingual bias requires a fundamental shift toward broad, multilingual exposure during the initial self-supervised pretraining phase, ensuring that the foundational codebooks of speech models are inherently universal rather than localized to a single dominant language.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_lingual_generalization_gap_monolin_c91671b9,
  title = {cross-lingual generalization gap (monolingual training bias)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-lingual-generalization-gap-monolingual-training-bias/rdr_c91671b997204f60828926e2585057e0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_lingual_generalization_gap_monolin_c91671b9_v1,
  title = {cross-lingual generalization gap (monolingual training bias)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-lingual-generalization-gap-monolingual-training-bias/rdr_c91671b997204f60828926e2585057e0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake