Overcoming Data Scarcity in End-to-End Speech Translation

30 concepts28 papers1 proposal

End-to-end (E2E) speech translation models offer an elegant promise: a single neural network that maps audio features in a source language directly to text or audio in a target language. By bypassing the traditional cascaded pipeline—which chains an Automatic Speech Recognition (ASR) model to a Machine Translation (MT) model—E2E systems avoid compounding errors across boundaries and can theoretically preserve acoustic nuances like prosody.

However, E2E models face a severe fundamental challenge: cross-modal data scarcity. While text-to-text translation corpora contain billions of sentences and ASR datasets span tens of thousands of hours, high-quality "triplet" data (audio paired with its direct translation) is exceptionally rare and expensive to curate. As Alexandre Bérard et al. observe in End-to-End Automatic Speech Translation of Audiobooks, this data starvation prevents high-capacity architectures from reaching their full potential.

To bridge this gap, researchers have developed techniques designed to inject the linguistic maturity of text models into speech architectures, align disparate modalities into shared semantic spaces, and synthetically scale training corpora.

The Modeling vs. Data Efficiency Trade-Off

Why do traditional cascaded systems historically outperform integrated E2E models? The answer lies in data efficiency. Cascaded architectures can independently leverage web-scale text data to train powerful language and translation components.

In a comprehensive survey, Matthias Sperber et al. formalized this tension in Speech Translation and the End-to-End Promise: Taking Stock of Where We Are. They argue that while E2E models possess superior theoretical modeling power, their reliance on scarce paired data often forces a practical return to modular or multi-task structures.

This data bottleneck manifests clearly in competitive benchmarks and low-resource deployment:

Cross-Modal Knowledge Distillation

If we cannot easily collect more speech-translation data, how can an E2E model acquire the linguistic robustness of a text-based machine translation model? The dominant solution is Knowledge Distillation (KD).

The intuition behind KD is to use a well-trained text-to-text MT model as a "teacher" and force the E2E speech translation model (the "student") to mimic its behavior. Instead of training the student solely on hard target labels (e.g., the correct translated text), the student is trained to match the teacher's output probability distributions (soft targets). These soft targets contain rich contextual information about word relationships and translation alternatives that text models learn from massive corpora.

[Text Input]   ---> [ MT Teacher Model ] ---> Soft Targets (Probabilities)
                                                     |
                                                     v (Minimize KL-Divergence)
[Audio Input]  ---> [Speech Student Model] -> Student Predictions

Hao Xiong et al. pioneered this approach for speech translation in End-to-End Speech Translation with Knowledge Distillation, demonstrating that transferring these probabilities directly bridges the cross-modal performance gap.

Marco Gaido expanded on this in Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?, showing that word-level distillation allows direct models to inherit linguistic maturity from text models, making their translations indistinguishable from human-preferred cascaded systems.

To maximize this transfer, Yikun Lei et al. introduced the CKDST framework in CKDST: Comprehensively and Effectively Distill Knowledge from Machine Translation to End-to-End Speech Translation. CKDST refines the distillation process by explicitly aligning speech and text representations within the encoder while decoupling target from non-target knowledge in the decoder, ensuring that the student model does not lose acoustic grounding while absorbing textual semantics.

Aligning the Modality Gap

Another way to circumvent data scarcity is to force the network to map audio features and text tokens into a modality-agnostic semantic space. If an encoder can project the spoken audio of a word and its written text counterpart into the exact same vector space, a decoder trained on abundant text-to-text translation data can seamlessly translate speech—even without direct speech-translation training pairs. This enables zero-shot speech translation.

Several strategies have been developed to enforce this cross-modal alignment:

Synthetic Scaling and Multi-Task Supervision

When real-world parallel data is fundamentally limited, researchers either manufacture synthetic data at scale or design architectures that decode multiple tasks sequentially to maximize the utility of existing data.

Pseudo-Labeling and Synthetic Corpora

Instead of waiting for human-curated datasets, high-performing text translation models can be used to generate synthetic target text for existing ASR audio. Rong Ye et al. scaled this approach in GigaST: A 10, 000-hour Pseudo Speech Translation Corpus by machine-translating the transcripts of a massive 10,000-hour ASR dataset, providing the volume of data necessary to train high-capacity E2E architectures.

This synthetic paradigm extends to speech-to-speech translation (S2ST), where data scarcity is even more acute due to the lack of parallel target audio. Sravya Popuri et al. leveraged self-supervised pre-training and data augmentation to treat discrete speech units like text in Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation.

More recently, Yu Pu et al. introduced the SynStard-1000 dataset in Empowering Large Language Models for End-to-End Speech Translation Leveraging Synthetic Data, demonstrating that fine-tuning lightweight Large Language Models on high-quality synthetic data allows direct models to finally surpass traditional cascaded baselines with remarkably low training time.

Consecutive and Decoupled Decoding

Rather than mapping audio directly to target text in a single black-box step, some architectures force a single decoder to handle intermediate representations.

Qianqian Dong et al. introduced this concept via the COSTT framework in Consecutive Decoding for Speech-to-text Translation. Their model uses a single decoder to generate the source transcript and the target translation consecutively from the same audio input:

$$\text{Audio Input} \longrightarrow \text{Decoder} \longrightarrow \underbrace{\text{Source Transcript}}_{\text{Task 1: ASR}} \longrightarrow \underbrace{\text{Target Translation}}_{\text{Task 2: MT}}$$

This design allows the model to implicitly leverage massive external MT datasets while avoiding the error propagation typical of traditional cascaded systems. The authors expanded on this multi-stage supervision in Listen, Understand and Translate: Triple Supervision Decouples End-to-end Speech-to-text Translation, decoupling the pipeline into acoustic encoding, semantic understanding, and translation blocks within a single end-to-end trainable network.

Similarly, Tomasz Potapczyk et al. utilized multi-task "dual-decoder" learning in Samsung's System for the IWSLT 2019 End-to-End Speech Translation Task to ensure that the shared representations remain grounded in both speech recognition and translation mechanics.

By combining these paradigms—distilling knowledge from text models, aligning cross-modal representations, and scaling with synthetic data—modern E2E speech translation has reached a tipping point. As Ngoc-Quan Pham et al. demonstrated in Effective combination of pretrained models - KIT@IWSLT2022, combining these data-stretching techniques with massive self-supervised pretrained models allows end-to-end architectures to finally overcome the data bottleneck and outperform traditional cascaded systems.

Key Foundational Work

A Comparison of Sequence-to-Sequence Models for Speech Recognition (2017) — contributed 2 concept elements to this direction.
End-to-End Automatic Speech Translation of Audiobooks (2018) — contributed 1 concept elements to this direction.
The USTC-NEL Speech Translation system at IWSLT 2018 (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_data_scarcity_be0f5dd2,
  title = {cross-modal data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-data-scarcity/rdr_be0f5dd2f9f84528bdab8716a2ef1b96},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_data_scarcity_be0f5dd2_v1,
  title = {cross-modal data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-data-scarcity/rdr_be0f5dd2f9f84528bdab8716a2ef1b96/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake