Textless Speech Translation with Discrete Acoustic Units

This direction is split into other directions: transcription dependency. Its last version stays readable and citable here.

Version 1 of 1166 papers182 conceptsfrozen 2026-10-04

For decades, the standard approach to speech translation followed a "cascade" architecture: first, transcribe audio into text (ASR), then translate that text into another language (MT), and finally synthesize the result back into speech (TTS). While effective for high-resource languages like English or Spanish, this pipeline hits a hard wall when faced with the nearly 3,000 languages that lack a standardized writing system.

Researchers are now bypassing text entirely. By discretizing continuous audio into "pseudo-text" units, new models can translate directly from speech to speech, bringing digital communication to oral-only cultures.

The Problem with the Text Bottleneck

The reliance on text creates a significant digital divide. As noted by Ann Lee and Peng‐Jen Chen in Direct speech-to-speech translation with discrete units, traditional systems exclude the 40% of global languages that are primarily oral. Even for languages with scripts, the cascade approach is prone to "error propagation"—if the ASR makes a mistake, the MT and TTS will confidently repeat it. Furthermore, text is a lossy representation; it strips away the prosody, emotion, and speaker identity inherent in the original voice.

Discretization: Turning Waves into Tokens

The core intuition behind "textless" translation is that speech, while continuous, is composed of a finite set of recurring phonetic patterns. If we can identify these patterns and assign them unique IDs, we can treat audio like a sequence of tokens—essentially creating a "synthetic script" for any language.

Several mechanisms have been proposed to achieve this:

  1. Self-Supervised Learning (SSL): Models like HuBERT or Wav2Vec 2.0 learn to cluster audio features into discrete units without any human labels. Ann Lee demonstrated that these units can serve as the target for translation, allowing models to predict audio tokens just as they would predict words in a sentence.
  2. Cross-Lingual VAEs: In the UWSpeech framework, Chen Zhang and Xu Tan used a Cross-Lingual Variational Autoencoder (XL-VAE) to "borrow" phonetic knowledge from written languages to discretize unwritten ones (UWSpeech: Speech to Speech Translation for Unwritten Languages).
  3. Visual Grounding: David Harwath and James R. Glass explored using images as an "interlingua." By showing a model a picture of a tree and playing audio of the word "tree" in two different languages, the model learns semantic equivalence without ever seeing the word "T-R-E-E" (Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech).

Unit-to-Unit Translation

Once speech is converted into a sequence of discrete units, the translation task becomes a sequence-to-sequence problem. A Transformer-based encoder-decoder model takes source units and predicts target units.

A major breakthrough in this area was the development of Translatotron 2 by Ye Jia and colleagues. Unlike its predecessor, it uses a single attention module to map speech directly to speech, significantly improving translation quality while robustly preserving the speaker's original voice (Translatotron 2: High-quality direct speech-to-speech translation with voice preservation).

For real-world unwritten languages like Taiwanese Hokkien, researchers at Meta AI used a "bridge" language (Mandarin) to facilitate training. By translating Hokkien speech into Mandarin text and then into Hokkien speech units, they built the first functional S2ST system for a language that lacks a standard script (Speech-to-Speech Translation for a Real-world Unwritten Language).

Evaluating Success Without Text

If a system doesn't produce text, how do we measure its accuracy? Traditional metrics like BLEU require a text transcript to compare against. To solve this, Mingda Chen and Justine Kao introduced BLASER, a text-free evaluation metric. BLASER uses multimodal embeddings to compare the "meaning" of the source audio and the translated audio directly, providing a way to benchmark systems for oral-only dialects (BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric).

Why It Works: The Power of Abstraction

The success of these models suggests that "text" is just one way to represent the underlying structure of language. By moving to discrete acoustic units, we are essentially discovering a more universal, "textless" representation of human speech. This allows models to:

  • Scale to low-resource languages by mining parallel audio from the web (SpeechMatrix).
  • Reduce Latency by avoiding the multi-step cascade process.
  • Preserve Nuance by keeping the translation process within the acoustic domain.

As Kevin Heffernan and Holger Schwenk argue, the future of equitable language technology depends on our ability to move beyond the "written word" and embrace the full complexity of the world's oral traditions (Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages).

Papers in this version

Showing 60 of 166 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_textless_speech_translation_with_discret_fd6baabe,
  title = {Textless Speech Translation with Discrete Acoustic Units},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textless-speech-translation-with-discrete-acoustic-units/rdr_fd6baabef24049339457e9e51d0966d2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_textless_speech_translation_with_discret_fd6baabe_v1,
  title = {Textless Speech Translation with Discrete Acoustic Units},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/textless-speech-translation-with-discrete-acoustic-units/rdr_fd6baabef24049339457e9e51d0966d2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}