Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models

Bajian Xiang, Shuaijiang Zhao, Tingwei Guo, Wei ZouEMNLP 2025

TL;DR

Large Speech Language Models (LSLMs) show a significant performance drop when processing speech compared to text, a phenomenon termed the "modality gap." This paper empirically investigates the underlying mechanisms of this gap, revealing that while speech and text representations align directionally, their magnitudes diverge. The authors propose an "Alignment Path Score" to quantify this token-level alignment, showing it strongly correlates with the modality gap and can be improved through targeted interventions.

What Problem Does This Paper Address?

End-to-end Large Speech Language Models (LSLMs) are revolutionizing human-computer interaction by directly processing both speech and text. However, they consistently underperform traditional pipeline systems (like ASR + LLM) on semantic understanding benchmarks. This performance disparity, known as the "modality gap," is substantial and warrants investigation. The authors aim to understand why this gap exists by analyzing the internal mechanisms of speech-text alignment within LSLMs and proposing solutions to bridge this gap.

Methods

The study employs a multi-faceted approach:

  • Empirical Analysis of Representations: They analyze speech and text representations at both coarse-grained (sequence-level) and fine-grained (token-level) depths within LSLMs.
  • Similarity Metrics: Cosine similarity and Euclidean distance are used to quantify the alignment between representations across layers.
  • Alignment Path Score (APS): A novel metric is introduced to measure token-level alignment by identifying the "alignment path" of speech frames to text tokens and quantifying the maximal similarity (or minimal distance) along this path.
  • Intervention Experiments: Targeted interventions are applied to speech token embeddings (angle projection and length normalization) to test their causal effect on improving speech input performance.
  • Reproducibility: The experiments are conducted across various LLM backbones and training strategies (full fine-tuning and LoRA) to ensure generalizability.

Datasets

  • Training Data: A synthesized dataset based on the Ke-Speech-Chat framework, comprising single-turn dialogue samples. Speech instructions were synthesized using CosyVoice and filtered based on WER from Whisper-large-v3. The dataset contains 637,283 samples, totaling 1,604 hours of speech.
  • Evaluation Data: Five subsets of the VoiceBench dataset (AdvBench, IfEval, OBQA, MMSU, and sd-qa), totaling 4,947 test samples, were used for evaluation. These subsets were chosen for their robustness and minimal reliance on additional LLMs.

Key Insights and Ideas

  • Direction vs. Magnitude: While speech and text representations become directionally aligned (higher cosine similarity) in deeper layers, their magnitudes diverge (higher Euclidean distance). This suggests LSLMs learn to map speech to text's semantic space but retain modality-specific magnitude characteristics.
  • Spontaneous Token-Level Alignment: LSLMs spontaneously learn a monotonic alignment pattern between speech frames and text tokens, even without explicit alignment objectives. This suggests the model inherently learns to map temporal speech structure to semantic text structure.
  • Alignment Path Score (APS) as a Predictor: The proposed APS metric shows a strong correlation with the modality gap, especially in LoRA-tuned models, indicating it's a reliable predictor of performance disparities.
  • Interventions Improve Performance: Targeted interventions, particularly angle projection to align speech token embeddings with text token embeddings, can significantly improve performance on challenging speech inputs, demonstrating a causal link between token-level alignment and accuracy.

Key Results

  • Modality Gap Magnitude: On average, LSLMs exhibit a 25% performance decline on speech inputs compared to text inputs. This is attributed to both reasoning capability erosion (8.79%) and suboptimal speech-text alignment (16.46%).
  • Correlation Strength: Cosine similarity showed a strong linear correlation with the modality gap (R²=0.75) under LoRA fine-tuning, while APS demonstrated even stronger correlations (R²=0.81 for cosine, R²=0.72 for Euclidean).
  • Intervention Effectiveness: Angle projection interventions improved performance in 6 out of 8 settings on the sd-qa subset, with Llama3.1-8B improving by 7.52% and Qwen2.5-7B by 4.19% when applied to the "Bottom3" least-aligned tokens. Length normalization was less effective.
  • Generalizability: The modality gap and alignment patterns were observed across various LLM backbones (LLaMA, Qwen) and training strategies (full fine-tuning, LoRA), as well as across different LSLM architectures (SpeechGPT, BLSP, GLM-4-Voice, Qwen2-Audio).

Limitations and Future Work

  • Generalizability: The experiments focused on a specific set of architectures and alignment frameworks. Broader validation on larger models and novel alignment strategies is needed.
  • Scope of Data and Tasks: The evaluation primarily used English, single-turn dialogues with synthetic speech. Future work should explore other languages, multi-turn conversations, and noisy real-world inputs.
  • Post-hoc Interventions: The current interventions are applied post-hoc at inference time. Integrating these insights directly into the training process to optimize cross-modal consistency and improve speech input performance is a key future direction.
  • Further Optimization: The study suggests that improving token-level alignment can enhance speech inference accuracy, paving the way for future advancements in LSLM development.

Summary updated

BibTeX
@inproceedings{xiang2025understanding,
  author = {Bajian Xiang and Shuaijiang Zhao and Tingwei Guo and Wei Zou},
  title = {{Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure