Textless Speech to Speech Translation via Discrete Units

This direction is merged into another direction: transcription dependency. Its last version stays readable and citable here.

Version 1 of 122 papers25 conceptsfrozen 2026-10-04

For decades, the standard approach to speech translation has been a "cascade": transcribe the source speech into text (ASR), translate that text into the target language (MT), and finally synthesize the result back into audio (TTS). While effective for high-resource languages like English or Spanish, this pipeline hits a hard wall for the nearly 3,000 languages that lack a standardized writing system. If you cannot write a language down, you cannot use it as an intermediate "middleman" for translation.

To solve this, researchers have moved toward textless speech-to-speech translation (S2ST). The core intuition is to treat speech not as a continuous waveform, but as a sequence of discrete acoustic units that function like a "pseudo-text."

The Discretization of Speech

The fundamental challenge of direct audio-to-audio translation is that raw audio is incredibly dense and noisy. A single second of audio contains thousands of samples, most of which represent speaker identity, background noise, or prosody rather than semantic meaning.

To strip away this noise, researchers like Ann Lee and Peng-Jen Chen proposed using self-supervised models (like HuBERT) to quantize speech into a finite vocabulary of discrete units. In Direct speech-to-speech translation with discrete units, they demonstrate that these units capture the underlying phonetic and semantic structure of speech while ignoring the specific "texture" of the speaker's voice.

By converting continuous audio into these tokens, the translation problem is transformed from a complex signal-processing task into a sequence-to-sequence problem—essentially, machine translation where the "words" are these learned acoustic units.

Bypassing the Text Barrier

The primary advantage of this "unit-to-unit" approach is its independence from orthography. In Speech-to-Speech Translation For A Real-world Unwritten Language, the Meta AI team applied this to Taiwanese Hokkien, a language primarily spoken rather than written.

Because there is no standard script for Hokkien, a traditional cascade model is impossible to train. Instead, the researchers used a "bridge" language (Mandarin) to align Hokkien audio with target speech units. This allowed the model to learn a direct mapping from source audio to target units without ever needing a single line of text. As Jeongsoo Choi notes in Textless Unit-to-Unit training, this also eliminates "error propagation"—the phenomenon where a single mistake in the transcription phase ruins the entire translation.

Preserving Nuance and Multimodality

Text is a lossy medium; it strips away the emotion, tone, and timing of the original speaker. By staying in the acoustic domain, textless models can theoretically preserve these paralinguistic features. Lu Lu and colleagues explored this in PolyVoice, treating S2ST as a language modeling problem that can preserve a speaker's original voice and style across the translation.

This "unit-based" philosophy extends beyond just audio. In AV-TranSpeech, Rongjie Huang integrated visual cues—specifically lip movements—directly into the discrete unit framework. Because the units represent phonetic building blocks, they can be mapped not just to audio, but to visual "lip" units. This enables "talking head translation," where a video of a speaker is translated into another language with perfectly synchronized mouth movements, as seen in the TransFace system.

Scaling Without Labels

The final piece of the puzzle is data. Traditional translation requires massive parallel corpora of text. To build textless systems, researchers have turned to "mining" parallel speech from the web. Paul-Ambroise Duquenne led the creation of SpeechMatrix, a large-scale corpus of mined speech-to-speech translations.

Furthermore, techniques like Discrete Unit Back-translation (DUB), introduced by Dong Zhang, allow models to learn from unlabeled audio. By translating target units back into the source domain and checking for consistency, models can improve their accuracy using only raw recordings, a vital capability for the world's many low-resource dialects.

By treating speech as its own "text," these researchers have moved translation from a text-centric paradigm to a truly multimodal one, ensuring that the digital divide does not leave the world's oral-only languages behind.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_textless_speech_to_speech_translation_vi_3657998b,
  title = {Textless Speech to Speech Translation via Discrete Units},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textless-speech-to-speech-translation-via-discrete-units/rdr_3657998bad834210b675f66276d27c7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_textless_speech_to_speech_translation_vi_3657998b_v1,
  title = {Textless Speech to Speech Translation via Discrete Units},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textless-speech-to-speech-translation-via-discrete-units/rdr_3657998bad834210b675f66276d27c7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}