Unsupervised ASR via Cross-Lingual Pseudo-Labeling

Tatiana Likhomanenko, Loren Lugosch, Ronan CollobertSubmission to ICLR 2024 · Rejected

TL;DR

Existing unsupervised automatic speech recognition (ASR) approaches avoid all labeled audio and often rely on complex GAN-based pipelines or phonetic lexicons. This paper introduces Cross-Lingual Pseudo-Labeling, a simple end-to-end method that bootstraps an acoustic model for a target language with zero labeled audio by pairing a source-language acoustic model with a target-language n-gram language model.

What Problem Does This Paper Address?

Most of the world's ~7,000 languages lack labeled audio datasets necessary to train modern supervised speech recognition systems. While "unsupervised ASR" aims to learn from unpaired speech audio and text alone, existing frameworks like wav2vec-U face two major hurdles:

  1. Model Complexity & Brittleness: They frequently rely on adversarial learning (GANs), silence-insertion heuristics, and heavy self-supervised pre-training.
  2. The Orthography Gap: Character-based unsupervised ASR historically lags far behind phoneme-based variants—yet low-resource languages rarely possess the pronunciation dictionaries or Grapheme-to-Phoneme (G2P) toolkits required for phoneme-level models.

The authors note that while target languages may lack transcribed audio, labeled audio does exist in high-resource languages (e.g., English, German). They ask: Can we use labeled audio from a source language to bootstrap an end-to-end, character-level ASR model in an unseen target language without adversarial training or phonetic lexicons?

Methods

The proposed framework relies on standard semi-supervised learning recipes (iterative pseudo-labeling) adapted to a cross-lingual transfer setting:

[Target Audio] ──> [Source Acoustic Model] ──> Logits
                                                 │
                                                 ▼
[Target n-gram LM] ──────────────────────> Beam-Search Decoding
                                                 │
                                                 ▼
                                        Constrained Pseudo-Labels (PLs)
                                                 │
                                                 ▼
                                 [Train/Update Target Acoustic Model]
  1. Model Setup: A 36-block Transformer acoustic model (AM, 255M parameters) trained via Connectionist Temporal Classification (CTC) on character tokens. A simple 4-gram target language model (LM) trained via KenLM on target text.
  2. Phase 1: Bootstrapping via Cross-Lingual IPL (Iterative Pseudo-Labeling):
  3. Initialize the target AM with the trained source AM.
  4. Run beam-search decoding on the target language's unlabeled audio, using the source AM outputs constrained by the target LM.
  5. Iteratively update the target AM on these pseudo-labels, re-generating labels every 4,000 iterations using the updated AM as teacher.
  6. Phase 2 (Optional refinement): slimIPL:
  7. Reinitialize from the source AM and fine-tune on Phase 1 pseudo-labels generated with a wider beam.
  8. Run LM-free online pseudo-labeling (slimIPL) with a cache of teacher pseudo-labels to refine acoustic boundaries and avoid LM overfitting.
  9. Cross-Alphabet Extension: When transferring between different scripts (e.g., Cyrillic to Latin), basic phonetic character transliteration rules map characters to a shared phonetic symbol space before training the source AM.

Datasets

  • Common Voice (v12.0): Paired and unpaired audio across Germanic (English, German), Romance (Spanish, French), Niger-Congo (Kinyarwanda, Swahili), and Afro-Asiatic (Hausa) languages, as well as Belarusian and Czech.
  • Common Crawl (CC-100): Used to extract text and train 4-gram target language models.
  • LJSpeech & LibriSpeech: Unpaired audio from LJSpeech (22.8h train split) and text from LibriSpeech LM corpus used to benchmark against wav2vec-U 2.0.

Key Insights and Ideas

  • Target LMs Fix Cross-Lingual Acoustic Confusion: Feeding Swahili audio into an English acoustic model yields high acoustic error (~98% WER zero-shot greedy). However, decoding with a Swahili n-gram LM acts as a strong regularizer, bringing zero-shot WER down to ~70%. During iterative pseudo-labeling, the acoustic model quickly aligns itself to native target phonetics.
  • Family Boundaries Aren't Hard Barriers: Cross-lingual PL transfers effectively within language families (e.g., German $\to$ English, French $\to$ Spanish), but surprisingly also across entirely disparate families (e.g., Indo-European English $\to$ Niger-Congo Swahili).
  • Simplicity Trumps Complex Pipelines: Standard self-training/pseudo-labeling on characters matches or beats complex adversarial unsupervised representations without requiring G2P tools or phoneme inventories.
  • Multilingual Sources Boost Bootstrapping: Combining multiple source languages (English + Spanish + French) into a single source AM provides richer cross-lingual acoustic coverage and uniformly improves downstream target performance.

Key Results

  • English to Swahili (en $\to$ sw):
  • Starting from a zero-shot baseline of 98.6% WER, Phase 1 Cross-Lingual PL achieves 27.1% greedy WER (20.0% with LM).
  • Phase 2 (slimIPL) brings performance down to 23.7% greedy WER and 18.0% with LM decoding (approaching the fully supervised Swahili baseline of 14.9% WER).
  • Comparison to wav2vec-U 2.0 on LJSpeech:
  • Transferring a German AM (de $\to$ en) via cross-lingual PL achieves 48.3% WER (pl-beam) using 800 hours of labeled German speech.
  • In contrast, character-based wav2vec-U 2.0 scores 64.0% WER while requiring 60,000 hours of unlabeled English speech representation pre-training (a 15.7% absolute WER improvement).
  • Cross-Alphabet Transfer (be $\to$ cs):
  • Bootstrapping Latin-based Czech using Cyrillic Belarusian (via character-level transliteration) achieves 62.6% greedy WER / 58.3% LM WER from an initial zero-shot >100% WER.

Limitations and Future Work

  • Data Volume Requirements: Cross-lingual PL requires a minimum threshold of unlabeled audio in the target language ($\ge 30$ hours) and a decent source AM (trained on 100–500+ hours). Performance was poor on Hausa due to only having 2.3 hours of target audio.
  • Vulnerability to Irregular Orthography: French consistently failed as a target language regardless of the source model. The authors hypothesize that French's deep orthography (silent letters, frequent liaisons, and irregular phoneme-to-grapheme mappings) prevents the character-level beam search from bootstrapping clean labels.
  • Future Work: Evaluating on wider language pairs, incorporating larger multilingual foundation models as source representations, and improving decoding strategies for irregular orthographies.

Summary updated

Abstract

Recent work has shown that it is possible to train an *unsupervised* automatic speech recognition (ASR) system using only unpaired audio and text. Existing unsupervised ASR methods assume that no labeled data can be used for training. We argue that even if one does not have any labeled audio for a given language, there is *always* labeled data available for other languages. We show that it is possible to use character-level acoustic models (AMs) from other languages to bootstrap an *unsupervised* AM in a new language. Here, ``unsupervised'' means no labeled audio is available for the *target* language. Our approach is based on two key ingredients: (i) generating pseudo-labels (PLs) of the *target* language using some *other* language AM and (ii) constraining these PLs with a *target language model*. Our approach is effective on Common Voice: e.g. transfer of English AM to Swahili achieves 18\% WER. It also outperforms character-based wav2vec-U 2.0 by 15\% absolute WER on LJSpeech with 800h of labeled German data instead of 60k hours of unlabeled English data.
BibTeX
@misc{likhomanenko2023unsupervised,
  author = {Tatiana Likhomanenko and Loren Lugosch and Ronan Collobert},
  title = {{Unsupervised ASR via Cross-Lingual Pseudo-Labeling}},
  year = {2023},
  eprint = {2305.13330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13330},
}
Figure