Cross Modal Pseudo Labeling for Speech Translation

This direction is merged into another direction: cascaded error propagation (cascaded system error propagation). Its last version stays readable and citable here.

Version 1 of 116 papers20 conceptsfrozen 2026-10-04

End-to-end speech translation (ST) models aim to map acoustic signals in one language directly to text in another. While conceptually elegant, these models face a massive data bottleneck: human-annotated triplets of audio, transcripts, and translations are far rarer than simple speech-to-transcript (ASR) or text-to-text (MT) data.

To overcome this, researchers have converged on a technique known as cross-modal pseudo-labeling. This approach synthesizes the missing links in the training data by using high-quality "teacher" models to generate labels for existing datasets, effectively turning the abundance of ASR and MT data into fuel for end-to-end systems.

The Textual Bridge

The core intuition behind pseudo-labeling is that while we lack speech-to-translation data, we have excellent models for the individual steps. If we have a dataset of speech and its corresponding transcripts (ASR data), we can use a pre-trained machine translation model to translate those transcripts into a target language.

As Jan Niehues and Danni Liu demonstrate in Maastricht University's Multilingual Speech Translation System for IWSLT 2021, this creates a synthetic "bridge." By pairing the original source audio with the machine-generated translation, we create a "pseudo-labeled" dataset that allows an end-to-end model to learn the direct mapping from sound to foreign text.

This technique is particularly potent because it transfers the vast knowledge contained in text-only translation models into the speech domain. Ngoc-Quan Pham and colleagues at KIT showed in Effective combination of pretrained models - KIT@IWSLT2022 that this "cross-task" labeling allows end-to-end models to finally outperform traditional cascaded systems (which transcribe then translate) by leveraging massive pretrained models like mBART50.

Scaling with Unlabeled Audio

The strategy extends beyond just repurposing ASR data. We can also leverage completely unlabeled audio—recordings without any transcripts at all.

In Self-Training for End-to-End Speech Translation, Juan Miguel Pino and his team utilized a teacher model (a cascade of ASR and MT) to generate synthetic target text for 56,000 hours of unlabeled audio. This massive injection of data led to gains of up to 8.3 BLEU points. The model isn't just memorizing the teacher; it is learning the underlying acoustic-to-semantic patterns from a volume of data that would be impossible to label by hand.

Synthesizing the Entire Pipeline

For more complex tasks like speech-to-speech translation (S2ST), the pseudo-labeling pipeline becomes more sophisticated. Qianqian Dong and Fengpeng Yue describe a "Pseudo Translation Labeling" (PTL) pipeline in Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation.

To create S2ST data from standard ASR datasets, they pass the source speech through:

  1. ASR to get the source text.
  2. MT to get the target text.
  3. TTS (Text-to-Speech) to generate target audio.

By training on these synthetic "speech-to-speech" pairs, the model learns to translate directly between audio waveforms, bypassing the need for intermediate text representations during inference.

Why Direct Translation Scales Better

A natural question is why we should bother with direct end-to-end models if we are using cascaded systems (ASR + MT) to generate the training data. Why not just use the cascade at runtime?

Recent evidence suggests that the intermediate transcription step in a cascade can act as a bottleneck. Oriol Pareras and Gerard I. Gállego found in Revisiting Direct Speech-to-Text Translation with Speech LLMs that while transcribing speech before translating it (Chain-of-Thought) is helpful for small datasets, direct translation scales much better. As the volume of pseudo-labeled training data increases, direct models continue to improve, eventually surpassing the performance of systems that rely on intermediate transcripts.

By using pseudo-labeling, we use the "bridge" of text to build a model that eventually learns to walk on its own, mapping sound to meaning without the constraints of a written transcript.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_pseudo_labeling_for_speech_t_06433602,
  title = {Cross Modal Pseudo Labeling for Speech Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-pseudo-labeling-for-speech-translation/rdr_06433602267544f4a71f47cbae534c37},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_pseudo_labeling_for_speech_t_06433602_v1,
  title = {Cross Modal Pseudo Labeling for Speech Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-pseudo-labeling-for-speech-translation/rdr_06433602267544f4a71f47cbae534c37/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}