The Modality Gap in Spoken Language Models

40 concepts25 papers

When large language models (LLMs) are trained on massive text corpora, they develop remarkable capabilities for abstract reasoning, logic, and factual retrieval. Naturally, as the field shifts toward "omni-modal" systems, we expect these cognitive strengths to seamlessly transfer across sensory inputs. If a model can solve a complex logic puzzle presented in text, it should theoretically solve the exact same puzzle when spoken aloud.

However, evaluating these models reveals a stark reality: transitioning from text to audio causes a severe drop in performance. This phenomenon—frequently termed the modality gap or intelligence degradation—means that direct audio processing often fractures the internal logic that makes LLMs so powerful in the first place.

Understanding why this gap exists, how it manifests, and how researchers are attempting to bridge it requires looking closely at the tension between continuous acoustic signals and discrete semantic reasoning.

The Illusion of Modality Invariance

Ideally, a model's internal reasoning engine should be decoupled from the specific input channel used to deliver information. Yet, cross-modal benchmarks consistently show that current architectures fail to treat audio as a first-class modality for complex reasoning.

As the authors of OmnixR demonstrate, frontier omni-modality models exhibit a profound lack of logical consistency across sensory channels. A system may successfully solve a problem when formatted as text, but fail or skip crucial reasoning steps entirely when the identical logic is delivered via audio. This cross-modal penalty is not an isolated quirk; Ke Wang et al. show in VoiceAssistant-Eval that even when the underlying visual context remains identical, changing the instruction modality from text to voice triggers a significant drop in model accuracy.

Furthermore, this disparity is unique to audio. While image-to-text capabilities have matured rapidly, Jinjie Ni et al. note in MixEval-X that audio-to-text understanding lags considerably behind. Testing models across tens of thousands of cross-modal permutations, Xingrui Wang et al. confirm in XModBench that internal representations are not yet modality-invariant, leading to severe performance drops when processing speech compared to text or visual inputs for the same semantic task.

The Cascaded vs. End-to-End Dilemma

To understand why direct audio processing degrades intelligence, it helps to contrast it with traditional cascaded pipelines. A cascaded system uses a separate Automatic Speech Recognition (ASR) model to transcribe audio into text, feeds that text to a standard LLM, and optionally passes the text output to a Text-to-Speech (TTS) engine.

Cascaded Pipeline:
[Audio Input] ──> (ASR) ──> [Discrete Text] ──> (Text LLM) ──> [High-Quality Reasoning]

End-to-End Spoken LLM:
[Audio Input] ────────────────────────────────> (SpeechLLM) ──> [Degraded Reasoning]

Cascaded pipelines preserve reasoning depth because they map the input back into the discrete token space where the LLM thrives. As Siddhant Arora et al. point out in On the Evaluation of Speech Foundation Models for Spoken Language Understanding, specialized text models can leverage vastly larger scales of linguistic data than integrated speech architectures. This explains why, as Anthony Kum Hoe Tung et al. show in SQuAD-SRC, end-to-end spoken question answering models struggle to outperform traditional cascades, suffering an average drop of 17.0 points in F1-score when forced to process audio directly rather than relying on clean text transcriptions.

Why, then, pursue end-to-end speech models at all? The answer lies in what text transcriptions leave behind. Cascades strip away all paralinguistic information—such as tone, emotion, volume, and speaker fidelity. The authors of VoxDialogue highlight this fundamental trade-off: text-based pipelines maximize reasoning depth but are blind to non-verbal cues, whereas direct audio models are highly sensitive to acoustic nuances but struggle with complex, long-context reasoning tasks.

When we bypass transcription and force a model to ingest raw audio tokens directly, the direct integration of audio features interferes with the established cognitive strengths of the underlying language model. The authors of SOVA-Bench document this interference clearly, showing that end-to-end speech-to-speech models experience a substantial reduction in general knowledge retrieval and logical deduction compared to their cascaded counterparts.

Why Audio Disrupts Internal Logic

What is it about audio that breaks an LLM's reasoning? The root cause lies in the fundamental difference between discrete text tokens and continuous acoustic signals. Text tokens are highly compressed, clean semantic units. Audio tokens, by contrast, are continuous, high-dimensional, and noisy.

When an end-to-end model processes speech, the acoustic complexity disrupts the model's internal semantic alignment. As Yuhao Zhang et al. explain in EchoX, the model's internal logic is fractured because the continuous nature of audio tokens distorts the structured semantic space developed during text pre-training.

This disruption becomes painfully obvious during multi-step reasoning. Text LLMs excel at Chain-of-Thought (CoT) prompting because they can generate a sequence of discrete intermediate tokens to anchor their logic. However, Wenqian Cui et al. discover in VoxEval that end-to-end spoken language models frequently experience performance drops when using audio-based Chain-of-Thought. Instead of helping, generating an extended stream of self-generated audio tokens causes the model to lose track of the required output format or become entirely incoherent.

Text Chain-of-Thought:
[Step 1: Discrete Idea] ──> [Step 2: Discrete Idea] ──> [Correct Answer]

Audio Chain-of-Thought:
[Continuous Audio] ──~> (Acoustic Drift/Noise) ──~> [Incoherent/Incorrect Answer]

This sensitivity to noise also cripples the model's ability to utilize external context. While text models handle Retrieval-Augmented Generation (RAG) exceptionally well, Heeseung Kim et al. demonstrate in Does Your Voice Assistant Remember? that voice interaction models are highly sensitive to recognition errors in retrieved text and frequently fail to integrate long-form evidence when prompted with speech. Furthermore, Marta R. Costa-jussà et al. show in 2M-BELEBELE that this modality gap is severely exacerbated in low-resource languages, where the lack of training data makes it even harder for the model to map continuous audio signals to stable semantic concepts.

Strategies for Aligning Speech and Text

To close the modality gap, researchers are developing training frameworks that force audio representations to align with the model's pre-existing text-based semantic space, without sacrificing acoustic sensitivity.

Interleaved Pre-training

One powerful approach is to introduce speech to the model in a way that mimics text structure. In InSerter, Dingdong WANG et al. propose an unsupervised pre-training strategy that interleaves synthesized speech segments directly into massive text corpora. By training the model to predict the subsequent text from a mixture of speech and text tokens, the model learns to treat spoken instructions with the same semantic weight as written prompts, narrowing the performance gap between speech and text inputs from 23.3% to just 1.3%.

Cross-Modal Distillation

Rather than training from scratch, another strategy borrows intelligence directly from a frozen text model. Santiago Cuervo et al. introduce the SALAD framework in Closing the Gap Between Text and Speech Understanding in LLMs. By using cross-modal distillation combined with an active selection process for synthetic data, they transfer the reasoning capabilities of a text-based "teacher" model into a speech-centric "student" model, achieving high-level reasoning while using 10x less data than standard approaches.

Hybrid Streaming Reasoning

Finally, instead of forcing the model to reason entirely within the volatile audio token space, researchers are exploring hybrid execution models. Yi-Jen Shih et al. address this in Can Speech LLMs Think while Listening? by allowing speech-centric models to generate text-based Chain-of-Thought tokens in the background while the user is still speaking. By anchoring the intermediate reasoning steps in discrete text tokens rather than continuous audio, this method bypasses acoustic drift entirely—yielding a 2.4x boost in reasoning accuracy while drastically reducing interaction latency.

Key Foundational Work

SQuAD-SRC: A Dataset for Multi-Accent Spoken Reading Comprehension (2023) — contributed 2 concept elements to this direction.
MuSLAM: Multitask, Multilingual Speech and Language Models (2023) — contributed 1 concept elements to this direction.
Effects of language modeling on speech-driven question answering (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modality_specific_performance_disparity_b3753a19,
  title = {modality-specific performance disparity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-specific-performance-disparity/rdr_b3753a19d852458db6b2d77eb3a026ec},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modality_specific_performance_disparity_b3753a19_v1,
  title = {modality-specific performance disparity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-specific-performance-disparity/rdr_b3753a19d852458db6b2d77eb3a026ec/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake