TL;DR
Existing unsupervised automatic speech recognition (ASR) approaches avoid all labeled audio and often rely on complex GAN-based pipelines or phonetic lexicons. This paper introduces Cross-Lingual Pseudo-Labeling, a simple end-to-end method that bootstraps an acoustic model for a target language with zero labeled audio by pairing a source-language acoustic model with a target-language n-gram language model.
What Problem Does This Paper Address?
Most of the world's ~7,000 languages lack labeled audio datasets necessary to train modern supervised speech recognition systems. While "unsupervised ASR" aims to learn from unpaired speech audio and text alone, existing frameworks like wav2vec-U face two major hurdles:
- Model Complexity & Brittleness: They frequently rely on adversarial learning (GANs), silence-insertion heuristics, and heavy self-supervised pre-training.
- The Orthography Gap: Character-based unsupervised ASR historically lags far behind phoneme-based variants—yet low-resource languages rarely possess the pronunciation dictionaries or Grapheme-to-Phoneme (G2P) toolkits required for phoneme-level models.
The authors note that while target languages may lack transcribed audio, labeled audio does exist in high-resource languages (e.g., English, German). They ask: Can we use labeled audio from a source language to bootstrap an end-to-end, character-level ASR model in an unseen target language without adversarial training or phonetic lexicons?
Methods
The proposed framework relies on standard semi-supervised learning recipes (iterative pseudo-labeling) adapted to a cross-lingual transfer setting:
[Target Audio] ──> [Source Acoustic Model] ──> Logits
│
▼
[Target n-gram LM] ──────────────────────> Beam-Search Decoding
│
▼
Constrained Pseudo-Labels (PLs)
│
▼
[Train/Update Target Acoustic Model]
- Model Setup: A 36-block Transformer acoustic model (AM, 255M parameters) trained via Connectionist Temporal Classification (CTC) on character tokens. A simple 4-gram target language model (LM) trained via KenLM on target text.
- Phase 1: Bootstrapping via Cross-Lingual IPL (Iterative Pseudo-Labeling):
- Initialize the target AM with the trained source AM.
- Run beam-search decoding on the target language's unlabeled audio, using the source AM outputs constrained by the target LM.
- Iteratively update the target AM on these pseudo-labels, re-generating labels every 4,000 iterations using the updated AM as teacher.
- Phase 2 (Optional refinement): slimIPL:
- Reinitialize from the source AM and fine-tune on Phase 1 pseudo-labels generated with a wider beam.
- Run LM-free online pseudo-labeling (
slimIPL) with a cache of teacher pseudo-labels to refine acoustic boundaries and avoid LM overfitting. - Cross-Alphabet Extension: When transferring between different scripts (e.g., Cyrillic to Latin), basic phonetic character transliteration rules map characters to a shared phonetic symbol space before training the source AM.
Datasets
- Common Voice (v12.0): Paired and unpaired audio across Germanic (English, German), Romance (Spanish, French), Niger-Congo (Kinyarwanda, Swahili), and Afro-Asiatic (Hausa) languages, as well as Belarusian and Czech.
- Common Crawl (CC-100): Used to extract text and train 4-gram target language models.
- LJSpeech & LibriSpeech: Unpaired audio from LJSpeech (22.8h train split) and text from LibriSpeech LM corpus used to benchmark against wav2vec-U 2.0.
Key Insights and Ideas
- Target LMs Fix Cross-Lingual Acoustic Confusion: Feeding Swahili audio into an English acoustic model yields high acoustic error (~98% WER zero-shot greedy). However, decoding with a Swahili n-gram LM acts as a strong regularizer, bringing zero-shot WER down to ~70%. During iterative pseudo-labeling, the acoustic model quickly aligns itself to native target phonetics.
- Family Boundaries Aren't Hard Barriers: Cross-lingual PL transfers effectively within language families (e.g., German $\to$ English, French $\to$ Spanish), but surprisingly also across entirely disparate families (e.g., Indo-European English $\to$ Niger-Congo Swahili).
- Simplicity Trumps Complex Pipelines: Standard self-training/pseudo-labeling on characters matches or beats complex adversarial unsupervised representations without requiring G2P tools or phoneme inventories.
- Multilingual Sources Boost Bootstrapping: Combining multiple source languages (English + Spanish + French) into a single source AM provides richer cross-lingual acoustic coverage and uniformly improves downstream target performance.
Key Results
- English to Swahili (
en$\to$sw): - Starting from a zero-shot baseline of 98.6% WER, Phase 1 Cross-Lingual PL achieves 27.1% greedy WER (20.0% with LM).
- Phase 2 (slimIPL) brings performance down to 23.7% greedy WER and 18.0% with LM decoding (approaching the fully supervised Swahili baseline of 14.9% WER).
- Comparison to wav2vec-U 2.0 on LJSpeech:
- Transferring a German AM (
de$\to$en) via cross-lingual PL achieves 48.3% WER (pl-beam) using 800 hours of labeled German speech. - In contrast, character-based wav2vec-U 2.0 scores 64.0% WER while requiring 60,000 hours of unlabeled English speech representation pre-training (a 15.7% absolute WER improvement).
- Cross-Alphabet Transfer (
be$\to$cs): - Bootstrapping Latin-based Czech using Cyrillic Belarusian (via character-level transliteration) achieves 62.6% greedy WER / 58.3% LM WER from an initial zero-shot >100% WER.
Limitations and Future Work
- Data Volume Requirements: Cross-lingual PL requires a minimum threshold of unlabeled audio in the target language ($\ge 30$ hours) and a decent source AM (trained on 100–500+ hours). Performance was poor on Hausa due to only having 2.3 hours of target audio.
- Vulnerability to Irregular Orthography: French consistently failed as a target language regardless of the source model. The authors hypothesize that French's deep orthography (silent letters, frequent liaisons, and irregular phoneme-to-grapheme mappings) prevents the character-level beam search from bootstrapping clean labels.
- Future Work: Evaluating on wider language pairs, incorporating larger multilingual foundation models as source representations, and improving decoding strategies for irregular orthographies.
Summary updated