End-to-end multimodal models—such as those translating speech directly to text or speech to speech—promise a elegant solution to the "telephone game" error propagation inherent in traditional cascaded systems (Sperber et al. in Speech Translation and the End-to-End Promise: Taking Stock of Where We Are). However, these models face a severe data bottleneck: while single-modality datasets (like text corpora or automated speech recognition data) are vast, high-quality parallel triplets across multiple modalities or low-resource languages are exceptionally rare.
To bridge this gap, researchers have turned to synthetic target generation and cross-modal pseudo-labeling. Instead of relying on scarce human annotations, this paradigm chains highly optimized, single-task models to generate proxy training targets.
Chaining Modalities via Pseudo-Labeling
The core mechanism of synthetic target generation relies on using a mature "teacher" model to fill in a missing modality or language layer. This effectively transforms abundant single-modality datasets into rich, multimodal training pairs.
For speech-to-text translation (ST), a common approach leverages large-scale Automatic Speech Recognition (ASR) datasets. Because these datasets only contain source audio and source transcripts, researchers use a machine translation (MT) system to generate synthetic target-language text. Tom Ko demonstrated the scalability of this approach by creating GigaST: A 10, 000-hour Pseudo Speech Translation Corpus, applying MT to ASR transcripts to bypass the prohibitive cost of manual human translation. Similarly, Ngoc-Quan Pham showed in Effective combination of pretrained models - KIT@IWSLT2022 that cross-task pseudo-labeling allows end-to-end models to successfully bootstrap from the combined expertise of independent recognition and translation modules.
[Source Audio] ---> (ASR Corpus Transcript) ---> [MT Model] ---> [Synthetic Target Text]
This pipeline can be extended symmetrically to more complex tasks like direct speech-to-speech translation (S2ST). In Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation, Qianqian Dong et al. introduced a Pseudo Translation Labeling pipeline that passes source speech through sequential ASR, MT, and text-to-speech (TTS) modules. This process synthesizes end-to-end training samples that allow models to learn direct acoustic-to-acoustic mappings without requiring human-labeled speech pairs.
The Intuition: Why Synthetic Targets Outperform Real Data
Intuitively, one might expect synthetic data to act as a noisy, second-best substitute for human ground truth. However, a recurring finding in recent literature is that training on synthetic targets can actually outperform training on authentic human data.
Oleksii Hrinchuk et al. explicitly demonstrated this "fake data advantage" in Leveraging Synthetic Targets for Machine Translation, showing that training models on targets generated by a larger teacher model consistently yields superior performance, particularly in data-scarce scenarios.
The reason this works boils down to knowledge distillation and structural regularization:
- Simplification of the Optimization Landscape: Human translations are highly diverse, containing stylistic variations, idioms, and colloquialisms. For an end-to-end model trying to map raw acoustic features to text, this high variance introduces significant noise. As Changhan Wang et al. observe in Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation, machine-generated pseudo-labels provide simpler, more deterministic, and linguistically consistent patterns. This acts as a regularizer, smoothing out the loss landscape for the student model.
- Alignment Over Naturalness: In multimodal settings, the structural alignment between inputs and targets is often more critical for model convergence than the absolute naturalness of the signal. Yexing Du et al. noted in Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion that high-quality synthetic signals generated by TTS engines serve as highly effective substitutes for human recordings because they preserve clean, predictable cross-modal correspondences.
Inverting the Pipeline: Source-Side Audio Synthesis
The paradigm is not limited to generating synthetic text targets; it works equally well in reverse by generating synthetic audio inputs to match abundant parallel text corpora.
Oleksii Hrinchuk et al. utilized this strategy in NVIDIA NeMo Offline Speech Translation Systems for IWSLT 2023 by applying TTS to the source side of text machine translation datasets. This method allows end-to-end speech models to inherit the vast vocabulary and structural diversity available in traditional text corpora.
Evaluating this approach in low-resource settings, Yasmin Moslem showed in Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation that source-side audio generation functions analogously to back-translation in text models, offering massive performance gains for languages like Irish where native speech-translation triplets are virtually non-existent.
Preserving Paralinguistic and Structural Context
A key challenge when moving from text to richer modalities is ensuring that non-textual features—such as emotion, timing, and speaker identity—are not lost during synthetic generation. Recent work addresses this by designing pipelines that explicitly preserve or project these features across domains.
- Acoustic and Emotional Invariance: To train cross-lingual emotion recognition models without native data, Fengmao Lv developed an emotion-preserving translation framework in Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages that synthesizes speech across target languages while strictly maintaining the vocal intensity and characteristics of the source audio.
- Temporal and Structural Alignment: For tasks like automatic dubbing, models must learn not just what to say, but when to say it. Derek Tam demonstrated in Isochrony-Aware Neural Machine Translation for Automatic Dubbing that timing-aware training data can be synthesized by using external alignment modules to project speech pauses from source audio directly onto generated target text sequences.
- Cross-Domain Timbre Adaptation: In tasks like speech-to-singing conversion, Mingda Liu showed in Bridging Speech and Singing: Multi-stage Speech-Prompted Singing Voice Conversion with Speaker Embedding Adaptation that generating proxy singing data by combining speech-derived timbre with singing content provides the necessary intermediate targets to stabilize cross-domain model training.
By treating generative models not merely as end-products but as infrastructure for data translation, synthetic target generation effectively decouples the difficulty of learning multimodal tasks. It allows models to bypass the lack of naturally occurring parallel data by learning from regularized, structurally aligned synthetic spaces.
Key Foundational Work
Speaker adaptation using a parallel phone set pronunciation dictionary for Thai-English bilingual TTS (2009) — contributed 2 concept elements to this direction.
Multimodal Neural Machine Translation for Low-resource Language Pairs using Synthetic Data (2018) — contributed 1 concept elements to this direction.
Using same-language machine translation to create alternative target sequences for text-to-speech synthesis (2009) — contributed 1 concept elements to this direction.