When humans engage in complex spoken dialogue, we do not simply map incoming acoustic waves directly to outgoing vocalizations. Instead, we perform an internal translation: we comprehend the words, organize our thoughts using a dense linguistic structure, plan our response, and finally articulate it.
Traditional end-to-end audio and speech language models often attempt to bypass this internal monologue, mapping audio features directly to target tasks or speech outputs. However, direct audio-to-audio or audio-to-task mapping frequently suffers from severe hallucinations, poor temporal logic, and an inability to follow multi-step instructions. To solve this, a powerful paradigm has emerged: cross-modal chain-of-thought (CoT), or textual reasoning streams. By forcing speech models to generate an intermediate text representation—such as a transcript, an emotional analysis, or a conceptual plan—before producing the final output, researchers are leveraging the deep, pre-trained reasoning capabilities of text-based Large Language Models (LLMs) to anchor auditory understanding.
Why Text Works: The Information Density Advantage
Why should a native audio model need text to think? The answer lies in information density and token efficiency. Raw audio signals and discrete speech tokens are highly sparse, high-dimensional, and cluttered with acoustic variations (such as pitch, duration, and background noise) that are irrelevant to pure logical reasoning. Text tokens, by contrast, are highly compressed, discrete semantic units optimized for abstract logic.
Shih et al. quantified this advantage in Can Speech LLMs Think while Listening?, demonstrating that reasoning within a text-based internal monologue is approximately twice as token-efficient as trying to reason using discrete speech tokens. The authors found that incorporating text-based reasoning traces into speech-centric architectures yields a 2.4x accuracy improvement compared to direct audio-to-audio processing. This confirms that even within native audio architectures, text remains a far more effective medium for logical "scratchpads."
Transcription as a Grounding Anchor
The most straightforward application of this concept is forcing a model to transcribe or describe what it hears before attempting a downstream task. This intermediate text stream acts as a semantic anchor, ensuring that subsequent reasoning remains strictly grounded in the literal content of the acoustic signal.
In Spoken Language Understanding (SLU), Do et al. showed in WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding that generating an audio transcription prior to semantic analysis prevents the model from losing track of the audio content and significantly reduces hallucinations during complex tasks like intent classification. Similarly, for joint speech recognition and extraction tasks, Li et al. demonstrated in Using Large Language Model for End-to-End Chinese ASR and NER that forcing a full transcription before performing Named Entity Recognition (NER) allows the model to resolve acoustic ambiguities using its internal linguistic knowledge.
This grounding mechanism is equally vital for non-speech audio. Large audio-language models frequently hallucinate the order or presence of sounds. To combat this, Kuan et al. proposed a multi-turn reasoning strategy called MATCH in Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning. The authors designed MATCH to force the model to describe an audio clip in detail before answering specific questions about it, ensuring the model forms a stable, grounded representation of the audio signal before executing discriminative tasks.
Sequential Modality Generation: Planning Before Speaking
When a speech language model must generate an audio response, the cross-modal chain-of-thought operates in reverse: the model plans its response in the text domain before initiating the complex task of acoustic generation. This is known as sequential modality generation.
Shen et al. explored this in Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation, noting that producing a text-based response before synthesizing speech tokens allows the model's reasoning capabilities to operate effectively in their native text domain. This strategy successfully avoids the "catastrophic forgetting" of reasoning skills often seen when fine-tuning LLMs for direct audio output. Similarly, Nachmani et al. demonstrated in Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM that determining the textual content of a response before deciding how to render it acoustically ensures high semantic accuracy.
In the context of end-to-end spoken dialogue systems, Arora developed a structured decoding strategy in Chain-of-Thought Training for Open E2E Spoken Dialogue Systems. The system generates an intermediate automatic speech recognition (ASR) transcript and a textual response before synthesizing the final audio output. This sequence acts as a semantic anchor that maintains logical coherence across the dialogue turn. The authors of UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice applied a similar cross-modal prompting strategy to speech-to-speech translation, showing that intermediate transcription and translation steps significantly improve translation fidelity compared to direct end-to-end mapping.
Advanced Scaffolding: Interleaved and Multi-Step Streams
Recent developments have extended these linear pipelines into more sophisticated, multi-step, and asynchronous reasoning streams that capture paralinguistic and contextual nuances.
For instance, cross-modal reasoning can be used to interpret how something is said. Sun et al. developed a framework in Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks that prompts a language model to explicitly reason through the significance of acoustic features (like hesitation or pitch) relative to textual content, translating raw vocal cues into logical conclusions about user intent. For empathetic dialogue, Xie et al. designed a structured pipeline in Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data that guides the model through an explicit sequence: transcription, emotion identification, and response generation.
To bridge the gap between complex textual reasoning and natural spoken language, Woo et al. introduced an intermediate "verbalization" step in Think, Verbalize, then Speak: Bridging Complex Thoughts and Comprehensible Speech. By training a model to interleave detailed reasoning steps with concise, speech-friendly summaries, they ensured the final spoken output remains natural without sacrificing deep reasoning.
Finally, in full-duplex conversational systems where latency is critical, Yao et al. demonstrated in FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Training that the text channel can act as an asynchronous predictive lead for the audio channel. This mimics human cognitive patterns where semantic intent is formed before speech is completed, ensuring that the model's verbal output is always grounded in a fully realized textual plan.
Key Foundational Work
Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation (2024) — contributed 2 concept elements to this direction.
Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM (2024) — contributed 2 concept elements to this direction.
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023) — contributed 2 concept elements to this direction.