For decades, the standard approach to speech translation has been a "cascade": transcribe the source speech into text (ASR), translate that text into the target language (MT), and finally synthesize the result back into audio (TTS). While effective for high-resource languages like English or Spanish, this pipeline hits a hard wall for the nearly 3,000 languages that lack a standardized writing system. If you cannot write a language down, you cannot use it as an intermediate "middleman" for translation.
To solve this, researchers have moved toward textless speech-to-speech translation (S2ST). The core intuition is to treat speech not as a continuous waveform, but as a sequence of discrete acoustic units that function like a "pseudo-text."
The Discretization of Speech
The fundamental challenge of direct audio-to-audio translation is that raw audio is incredibly dense and noisy. A single second of audio contains thousands of samples, most of which represent speaker identity, background noise, or prosody rather than semantic meaning.
To strip away this noise, researchers like Ann Lee and Peng-Jen Chen proposed using self-supervised models (like HuBERT) to quantize speech into a finite vocabulary of discrete units. In Direct speech-to-speech translation with discrete units, they demonstrate that these units capture the underlying phonetic and semantic structure of speech while ignoring the specific "texture" of the speaker's voice.
By converting continuous audio into these tokens, the translation problem is transformed from a complex signal-processing task into a sequence-to-sequence problem—essentially, machine translation where the "words" are these learned acoustic units.
Bypassing the Text Barrier
The primary advantage of this "unit-to-unit" approach is its independence from orthography. In Speech-to-Speech Translation For A Real-world Unwritten Language, the Meta AI team applied this to Taiwanese Hokkien, a language primarily spoken rather than written.
Because there is no standard script for Hokkien, a traditional cascade model is impossible to train. Instead, the researchers used a "bridge" language (Mandarin) to align Hokkien audio with target speech units. This allowed the model to learn a direct mapping from source audio to target units without ever needing a single line of text. As Jeongsoo Choi notes in Textless Unit-to-Unit training, this also eliminates "error propagation"—the phenomenon where a single mistake in the transcription phase ruins the entire translation.
Preserving Nuance and Multimodality
Text is a lossy medium; it strips away the emotion, tone, and timing of the original speaker. By staying in the acoustic domain, textless models can theoretically preserve these paralinguistic features. Lu Lu and colleagues explored this in PolyVoice, treating S2ST as a language modeling problem that can preserve a speaker's original voice and style across the translation.
This "unit-based" philosophy extends beyond just audio. In AV-TranSpeech, Rongjie Huang integrated visual cues—specifically lip movements—directly into the discrete unit framework. Because the units represent phonetic building blocks, they can be mapped not just to audio, but to visual "lip" units. This enables "talking head translation," where a video of a speaker is translated into another language with perfectly synchronized mouth movements, as seen in the TransFace system.
Scaling Without Labels
The final piece of the puzzle is data. Traditional translation requires massive parallel corpora of text. To build textless systems, researchers have turned to "mining" parallel speech from the web. Paul-Ambroise Duquenne led the creation of SpeechMatrix, a large-scale corpus of mined speech-to-speech translations.
Furthermore, techniques like Discrete Unit Back-translation (DUB), introduced by Dong Zhang, allow models to learn from unlabeled audio. By translating target units back into the source domain and checking for consistency, models can improve their accuracy using only raw recordings, a vital capability for the world's many low-resource dialects.
By treating speech as its own "text," these researchers have moved translation from a text-centric paradigm to a truly multimodal one, ensuring that the digital divide does not leave the world's oral-only languages behind.