For decades, the standard way to build a machine that "talks" has been a relay race. First, an Automatic Speech Recognition (ASR) module transcribes audio into text. Next, a Large Language Model (LLM) processes that text to generate a response. Finally, a Text-to-Speech (TTS) engine reads that response aloud.
This "cascaded" architecture is intuitive, but it suffers from a fundamental flaw: the textual bottleneck. When we convert a rich, multi-dimensional audio signal into a flat string of characters, we discard almost everything that makes human speech human. As Wenqian Cui and colleagues argue in Recent Advances in Speech Language Models: A Survey, this process strips away paralinguistic cues—pitch, rhythm, emotion, and speaker identity—that are essential for true understanding.
Researchers are now moving toward "textless" or end-to-end Spoken Language Models (SpeechLMs) that treat audio as a primary modality, rather than a mere proxy for text.
The Cost of Transcription
In a cascaded system, the LLM is "deaf." It only sees the words. If a user says "That's great" with heavy sarcasm, the ASR output is simply "That's great." The model loses the tonal shift that signals the opposite meaning.
This loss of expressive nuance is a recurring theme across the literature. Bin Wang and the team behind AudioBench: A Universal Benchmark for Audio Large Language Models note that cascaded systems cannot capture non-verbal information like background environmental sounds or speaker accents. Similarly, Gyuwan Kim and Minjeong Kim highlight in ST-BERT that intent recognition often fails because expressive nuances like prosody are lost before semantic analysis even begins.
Beyond information loss, cascades introduce cumulative latency. Each stage must finish before the next begins, making real-time, "full-duplex" conversation—where humans naturally interrupt or offer backchannels like "mm-hmm"—nearly impossible.
Bypassing Text with Speech Tokens
The emerging solution is to represent speech using discrete "tokens," much like how we represent text. Instead of transcribing audio into letters, models like SpiRit-LM, developed by Tu Anh Nguyen and Benjamin Müller, interleave text tokens with speech tokens in a single, unified stream.
This allows the model to learn the relationship between the sound and the meaning directly. By training on massive amounts of data—such as the 1 trillion tokens used by Aohan Zeng in Scaling Speech-Text Pre-training with Synthetic Interleaved Data—these models develop a "cross-modal" understanding. They can reason about a spoken question and generate a spoken answer without ever needing an intermediate text file.
Capturing the "How" of Speech
When a model "hears" the audio directly, it can access paralinguistic features that were previously invisible.
- Emotion and Pitch: Heeseung Kim and Kyeongseok Jeong demonstrated in their work on Unified Spoken Dialog Models (USDM) that semantic speech tokens naturally contain rich emotional information. This allows the AI to generate responses that are prosodically appropriate—matching the user's energy or tone.
- Speaker Identity: One of the most striking examples of this direction is Translatotron. Jia Ye and Fadi Biadsy showed that by translating speech-to-speech directly, the model can retain the original speaker’s voice and tempo in the target language—a feat impossible for text-based translators.
- Visual Context: The frontier is moving even beyond audio. Se Jin Park and the team at KAIST introduced AV-EmoDialog, which processes raw audio-visual inputs. By "seeing" facial expressions alongside "hearing" tone, the model achieves a level of emotional resonance that cascaded, text-only systems cannot replicate.
Toward Fluid Interaction
The ultimate goal of this research is to move AI away from the rigid "your turn, my turn" interaction of chatbots.
In Real-Time Textless Dialogue Generation, Julie Carson-Berndsen describes a system that predicts conversational actions every 160ms. By ditching the three-step pipeline, the model can handle overlaps and interruptions in real-time. Similarly, Zuwei Long and Yunhang Shen introduced VITA-Audio, which uses a "zero-delay" generation module to achieve a 3–5x speedup over previous models.
By eliminating the textual bottleneck, researchers are not just making AI faster; they are making it capable of understanding the full spectrum of human communication. As Shengpeng Ji notes in WavChat, we are moving toward a world where AI communicates with the same fluidity and expressive depth as a human conversation partner.