Unified End to End Speech to Speech Translation

This direction is merged into another direction: cascaded error propagation (cascaded system error propagation). Its last version stays readable and citable here.

Version 1 of 133 papers33 conceptsfrozen 2026-10-04

For decades, the standard approach to translating spoken language has been a "cascade" of independent systems: first, Automatic Speech Recognition (ASR) transcribes the audio to text; next, Machine Translation (MT) converts that text into a target language; finally, Text-to-Speech (TTS) synthesizes the result back into audio.

While modular, this pipeline has two fundamental flaws: latency and error propagation. In a cascade, each stage must wait for the previous one to finish, creating a lag that makes real-time conversation feel disjointed. More critically, as Michelle Tadmor Ramanovich and Ye Jia explain in CVSS Corpus and Massively Multilingual Speech-to-Speech Translation, errors in the initial transcription phase are compounded and amplified by the translation and synthesis modules, leading to a "broken telephone" effect where the final output is significantly degraded.

The research direction emerging across these papers is the move toward Direct or End-to-End (E2E) Speech-to-Speech Translation (S2ST)—a unified architecture that maps source audio directly to target audio.

Collapsing the Pipeline

The core intuition behind E2E models is to treat speech translation as a single, holistic task rather than a sequence of sub-tasks. By bypassing intermediate text representations, these models can theoretically preserve paralinguistic information—like tone, emotion, and speaker identity—that is typically lost in transcription.

In Translatotron 2: Robust direct speech-to-speech translation, Michelle Tadmor Ramanovich and colleagues demonstrate a model that uses a single attention module to bridge the gap. This approach not only reduces the "error compounding" seen in cascades but also enables the preservation of the original speaker's voice across languages, a feat difficult to achieve when the system only "sees" text in the middle.

From Spectrograms to Discrete Units

A major hurdle for direct S2ST is that audio is continuous and high-dimensional, making it much harder to model than discrete text tokens. A breakthrough mechanism in this field is the use of self-supervised discrete acoustic units.

Instead of predicting a complex spectrogram, models like those proposed by Ann Lee et al. in Direct speech-to-speech translation with discrete units convert target speech into a sequence of discrete tokens (quantized features from models like HuBERT). This allows the translation task to be framed as a generative language modeling problem. Lu Lu and Qianqian Dong take this further with PolyVoice, treating S2ST as a pure language modeling task using a GPT-style decoder to generate both semantic and acoustic units.

Overcoming Data Scarcity

Direct S2ST models require massive amounts of parallel speech-to-speech data, which is far rarer than text-to-text data. Researchers have developed several strategies to bridge this "data gap":

  1. Knowledge Distillation: Marco Gaido and Matteo Negri in End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020 show how an E2E model can "learn" from a pre-trained machine translation teacher, essentially distilling the teacher's linguistic knowledge into the speech encoder.
  2. Curriculum Pre-training: Chengyi Wang and Shujie Liu argue in Curriculum Pre-training for End-to-End Speech Translation that the task is too "heavy" for a model to learn at once. They propose a curriculum that starts with simple transcription, moves to contextual understanding, and finally tackles full translation.
  3. Pseudo-Labeling: In Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation, Qianqian Dong and Tom Ko use a high-quality cascade system to generate synthetic speech-to-speech labels for unlabelled audio, which is then used to train a more efficient direct model.

Simultaneous Translation and Real-Time Interaction

The ultimate goal of this research is seamless, real-time interpretation. Systems like RealTranS, developed by Liangyou Li and Qun Liu in RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer, introduce "Wait-K" strategies. These allow the model to begin translating before the speaker has even finished their sentence, drastically reducing latency.

More recently, Shaolei Zhang and Qingkai Fang introduced StreamSpeech, an "all-in-one" model that jointly learns translation and timing. By using non-autoregressive generation, as seen in NAST-S2x, these systems can achieve up to a 28x speedup in decoding, bringing us closer to the science-fiction ideal of a universal translator.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_end_to_end_speech_to_speech_tran_8d826e05,
  title = {Unified End to End Speech to Speech Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-end-to-end-speech-to-speech-translation/rdr_8d826e05c16746ddb6f188545eb19054},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_end_to_end_speech_to_speech_tran_8d826e05_v1,
  title = {Unified End to End Speech to Speech Translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-end-to-end-speech-to-speech-translation/rdr_8d826e05c16746ddb6f188545eb19054/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}