Synthetic Target Generation for Multimodal Translation

61 concepts50 papers1 proposal

End-to-end multimodal models—such as those translating speech directly to text or speech to speech—promise a elegant solution to the "telephone game" error propagation inherent in traditional cascaded systems (Sperber et al. in Speech Translation and the End-to-End Promise: Taking Stock of Where We Are). However, these models face a severe data bottleneck: while single-modality datasets (like text corpora or automated speech recognition data) are vast, high-quality parallel triplets across multiple modalities or low-resource languages are exceptionally rare.

To bridge this gap, researchers have turned to synthetic target generation and cross-modal pseudo-labeling. Instead of relying on scarce human annotations, this paradigm chains highly optimized, single-task models to generate proxy training targets.

Chaining Modalities via Pseudo-Labeling

The core mechanism of synthetic target generation relies on using a mature "teacher" model to fill in a missing modality or language layer. This effectively transforms abundant single-modality datasets into rich, multimodal training pairs.

For speech-to-text translation (ST), a common approach leverages large-scale Automatic Speech Recognition (ASR) datasets. Because these datasets only contain source audio and source transcripts, researchers use a machine translation (MT) system to generate synthetic target-language text. Tom Ko demonstrated the scalability of this approach by creating GigaST: A 10, 000-hour Pseudo Speech Translation Corpus, applying MT to ASR transcripts to bypass the prohibitive cost of manual human translation. Similarly, Ngoc-Quan Pham showed in Effective combination of pretrained models - KIT@IWSLT2022 that cross-task pseudo-labeling allows end-to-end models to successfully bootstrap from the combined expertise of independent recognition and translation modules.

[Source Audio] ---> (ASR Corpus Transcript) ---> [MT Model] ---> [Synthetic Target Text]

This pipeline can be extended symmetrically to more complex tasks like direct speech-to-speech translation (S2ST). In Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation, Qianqian Dong et al. introduced a Pseudo Translation Labeling pipeline that passes source speech through sequential ASR, MT, and text-to-speech (TTS) modules. This process synthesizes end-to-end training samples that allow models to learn direct acoustic-to-acoustic mappings without requiring human-labeled speech pairs.

The Intuition: Why Synthetic Targets Outperform Real Data

Intuitively, one might expect synthetic data to act as a noisy, second-best substitute for human ground truth. However, a recurring finding in recent literature is that training on synthetic targets can actually outperform training on authentic human data.

Oleksii Hrinchuk et al. explicitly demonstrated this "fake data advantage" in Leveraging Synthetic Targets for Machine Translation, showing that training models on targets generated by a larger teacher model consistently yields superior performance, particularly in data-scarce scenarios.

The reason this works boils down to knowledge distillation and structural regularization:

  1. Simplification of the Optimization Landscape: Human translations are highly diverse, containing stylistic variations, idioms, and colloquialisms. For an end-to-end model trying to map raw acoustic features to text, this high variance introduces significant noise. As Changhan Wang et al. observe in Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation, machine-generated pseudo-labels provide simpler, more deterministic, and linguistically consistent patterns. This acts as a regularizer, smoothing out the loss landscape for the student model.
  2. Alignment Over Naturalness: In multimodal settings, the structural alignment between inputs and targets is often more critical for model convergence than the absolute naturalness of the signal. Yexing Du et al. noted in Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion that high-quality synthetic signals generated by TTS engines serve as highly effective substitutes for human recordings because they preserve clean, predictable cross-modal correspondences.

Inverting the Pipeline: Source-Side Audio Synthesis

The paradigm is not limited to generating synthetic text targets; it works equally well in reverse by generating synthetic audio inputs to match abundant parallel text corpora.

Oleksii Hrinchuk et al. utilized this strategy in NVIDIA NeMo Offline Speech Translation Systems for IWSLT 2023 by applying TTS to the source side of text machine translation datasets. This method allows end-to-end speech models to inherit the vast vocabulary and structural diversity available in traditional text corpora.

Evaluating this approach in low-resource settings, Yasmin Moslem showed in Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation that source-side audio generation functions analogously to back-translation in text models, offering massive performance gains for languages like Irish where native speech-translation triplets are virtually non-existent.

Preserving Paralinguistic and Structural Context

A key challenge when moving from text to richer modalities is ensuring that non-textual features—such as emotion, timing, and speaker identity—are not lost during synthetic generation. Recent work addresses this by designing pipelines that explicitly preserve or project these features across domains.

By treating generative models not merely as end-products but as infrastructure for data translation, synthetic target generation effectively decouples the difficulty of learning multimodal tasks. It allows models to bypass the lack of naturally occurring parallel data by learning from regularized, structurally aligned synthetic spaces.

Key Foundational Work

Speaker adaptation using a parallel phone set pronunciation dictionary for Thai-English bilingual TTS (2009) — contributed 2 concept elements to this direction.
Multimodal Neural Machine Translation for Low-resource Language Pairs using Synthetic Data (2018) — contributed 1 concept elements to this direction.
Using same-language machine translation to create alternative target sequences for text-to-speech synthesis (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_synthetic_target_generation_de64ec4c,
  title = {synthetic target generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-target-generation/rdr_de64ec4cd91f4e62a03b79cb24742c2f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_synthetic_target_generation_de64ec4c_v1,
  title = {synthetic target generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-target-generation/rdr_de64ec4cd91f4e62a03b79cb24742c2f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake