Traditional speech translation systems rely on a three-step cascade: first, transcribe spoken audio into text via Automatic Speech Recognition (ASR); second, translate that text into the target language using Machine Translation (MT); and third, synthesize the translated text back into audio via Text-to-Speech (TTS).
While this pipeline works well for high-resource languages, it hits a fundamental wall when applied globally. Roughly 40% of the world’s languages are unwritten or lack a standardized orthography, as noted by Peng-Jen Chen et al. in Speech-to-Speech Translation for a Real-world Unwritten Language. Furthermore, text-centric models inherently exclude communities with low literacy rates or strong oral traditions—a challenge highlighted by Nathaniel D. Robinson et al. in Kreyòl-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages and Kevin Assogba in Bayelemabaga: Creating Resources for Bambara NLP.
To bridge this digital divide, a new paradigm has emerged: textless speech-to-speech translation (S2ST). By bypassing intermediate textual representations entirely, these systems process and generate speech directly, unlocking translation capabilities for purely oral languages.
Discretizing the Waveform: From Audio to Speech Tokens
The core challenge of textless translation is structural. Text is naturally discrete—composed of distinct letters, subwords, or words—making it easy to model using standard sequence-to-sequence architectures. Raw audio, by contrast, is a continuous, high-dimensional waveform packed with acoustic variance.
To apply modern language modeling techniques directly to audio, we must first convert continuous speech into a sequence of discrete "speech tokens." This is achieved through self-supervised representation learning and quantization.
Continuous Audio Waveform
│
▼ (Self-Supervised Encoder)
Continuous Feature Vectors
│
▼ (Vector Quantization / Codebook)
Discrete Acoustic Units (e.g., [42, 107, 19, 19, 88])
Using architectures like Vector Quantized Variational Autoencoders (VQ-VAEs), models can automatically discover a finite "vocabulary" of acoustic units from raw audio without any human transcription. Sakriani Sakti et al. demonstrated this feasibility in Speech-to-speech Translation between Untranscribed Unknown Languages, proving that stable symbolic representations can be extracted purely from acoustic patterns.
Once speech is represented as a sequence of discrete units, translation becomes a Unit-to-Unit Translation (UTUT) problem. As explored by Minsu Kim et al. in Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation, these discrete tokens can be fed into standard sequence-to-sequence networks. This concept can be pushed even further by treating speech units identically to text tokens within generative language models. For instance, Tom Ko developed PolyVoice: Language Models for Speech to Speech Translation, utilizing a decoder-only architecture to predict target speech tokens autoregressively, achieving high-fidelity translation and voice preservation without a single string of text.
Disentangling Meaning from Sound
While discretizing audio solves the structural problem, it introduces a semantic one: acoustic multimodality. Text only captures the linguistic message. A speech waveform, however, simultaneously encodes what is being said (semantics) and how it is being said (speaker identity, pitch, rhythm, and emotional tone).
If a translation model attempts to map raw acoustic units directly from one language to another, it risks confounding these channels. For example, the model might mistakenly try to translate a speaker's unique vocal pitch or background noise into a semantic change in the target language.
To prevent this, textless architectures must actively disentangle linguistic content from non-linguistic features. Rongjie Huang et al. addressed this in TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation by introducing a "Bilateral Perturbation" technique. By intentionally stripping away variance in pitch and duration during training, they forced the translation network to focus exclusively on the invariant semantic core of the audio.
Another strategy involves anchoring the unwritten language to a shared cross-lingual space. In UWSpeech: Speech to Speech Translation for Unwritten Languages, Chen Zhang et al. utilized a Cross-Lingual Variational Autoencoder (XL-VAE). The network learns to project continuous speech from an unwritten language into a discrete space bound to the phonetic properties of a well-resourced target language, ensuring that the discovered units track meaningful phonetic boundaries rather than acoustic noise.
Overcoming Data Scarcity via Visual Grounding
How do we train a translation model when there are no parallel text corpora available to guide it? For unwritten languages, collecting millions of hours of parallel audio-to-audio data is practically impossible.
One elegant solution is to use the physical world as an intermediate representation—specifically through visual grounding. While two languages may share no written scripts or phonetic overlap, they share the same visual reality. An image of a tree looks the same whether the speaker describes it in English or an unwritten oral dialect.
Galen Chuang et al. pioneered this intuition in Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech, demonstrating that models can learn semantic embeddings of untranscribed speech by aligning audio waveforms with corresponding images. The visual scene acts as a universal "Rosetta Stone."
Building on this, David Harwath et al. showed in Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio that by mapping speech from two different languages into a shared visual embedding space, the model can automatically discover "pseudo-words" and cluster their semantic translations without any textual supervision.
Language A Spoken Audio ───► [ Shared Visual ] ◄─── Language B Spoken Audio
[ Embedding Space]
▲
│
Target Image
Alternatively, models can leverage acoustic features transferred from high-resource speech tasks. Adam Lopez demonstrated in Pre-training on high-resource speech recognition improves low-resource speech-to-text translation that encoders pre-trained on high-resource ASR tasks learn generalized phonetic features. These features transfer effectively to low-resource translation contexts, minimizing the volume of direct parallel data required to achieve comprehension.
Evaluating Speech Without Scripts
A persistent paradox in textless speech translation research has been evaluation. Even when a model successfully bypasses text during training and inference, researchers traditionally evaluate its performance by passing the output audio through an auxiliary ASR system to calculate text-based metrics like BLEU. As noted by Ann Lee et al. in Textless Speech-to-Speech Translation on Real Data, this reliance on text-based proxies creates an artificial bottleneck and fails entirely for languages that lack an ASR tool or a standardized script.
To close this loop, the field has developed purely acoustic evaluation frameworks. The authors of BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric introduced a text-free metric that operates directly on speech signals. By leveraging multimodal speech-text embeddings, BLASER computes semantic similarity scores directly between audio clips, enabling objective evaluation for oral-only dialects.
With text-free evaluation and tokenization firmly established, advanced alignment techniques can now optimize speech models directly in the audio domain. For example, Guan-Ting Lin et al. developed Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback. This framework applies Direct Preference Optimization (DPO) directly to spoken language models using acoustic feedback loops, refining semantic coherence and translation quality without ever relying on intermediate text transcriptions.