Measuring the Redundancy of Decoder Layers in SpeechLLMs

Adel Moumen, Guangzhi Sun, Philip C. Woodland

In the world of AI, bigger is often better—until you have to pay the cloud bill or wait for a slow inference. Speech Large Language Models (SpeechLLMs) typically bolt a speech encoder onto a massive LLM decoder, with the latter often accounting for over 90% of the total parameters. But does a model really need 8 billion parameters just to transcribe "Hello, how are you?"

A new paper from researchers at the University of Cambridge, titled "Measuring the Redundancy of Decoder Layers in SpeechLLMs," suggests the answer is a resounding "no." By systematically pruning these models, they discovered that a significant portion of the decoder is essentially dead weight inherited from the original text-based LLM.

TL;DR

SpeechLLMs are massively over-parameterized for speech tasks, with up to 40% of decoder layers being redundant. This redundancy is largely inherited from the base LLM and can be identified using simple text-only tests, allowing for significant speedups and memory savings without sacrificing performance.

What Problem Does This Paper Address?

Modern SpeechLLMs (like the SLAM framework) work by "routing" speech features into a pre-trained LLM decoder. While these models achieve state-of-the-art results in Automatic Speech Recognition (ASR) and Translation (AST), they are computationally expensive.

Historically, speech tasks were handled by much smaller, specialized models. This creates a massive gap: why do we need a 7B or 8B parameter decoder for a task that used to require a fraction of that? The authors set out to quantify this "excess capacity" and determine if we can prune these models to make them faster and leaner.

Methods

The researchers employed a three-step approach to identify and remove redundancy:

  1. Redundancy Proxy: They used angular distance between hidden states as a metric. If the representation of data at Layer $L$ is almost geometrically identical to the representation at Layer $L+n$, the layers in between aren't doing much transformative work.
  2. Optimal Pruning Path: Instead of removing layers at random, they calculated an "optimal path" that minimizes the angular distance for a given number of removed layers.
  3. Post-Pruning Healing: Removing layers creates a "jump" in the model's logic. To fix this, they used LoRA (Low-Rank Adaptation) to fine-tune the remaining decoder layers and the "projector" (the bridge between speech and text) for a few thousand iterations. This "healing" process is what allows the model to recover its performance.

Datasets

The study was comprehensive, testing across two model families (Qwen2.5 and Llama 3.1/3.2) at three scales (1B to 8B).

  • ASR Evaluation: LibriSpeech (clean and noisy) and the diverse Loquacious dataset.
  • Translation Evaluation: CoVoST2 (English→German and French→English).

Key Insights and Ideas

  • Redundancy is an Inheritance: One of the most striking findings is that the "map" of redundant layers is nearly identical whether the model is processing text or speech. This means the redundancy isn't caused by the speech task itself; it's a property of the pre-trained LLM. The "aha" moment: You can identify which layers to cut using cheap text-only forward passes without ever touching a GPU-heavy speech dataset.
  • The "Bigger is Redundant-er" Rule: Larger models are more prunable. An 8B model can lose ~40% of its layers and still perform well, whereas a 1B model starts to break much sooner. Larger models have more "excess capacity" to spare.
  • Joint Healing is Non-Negotiable: Pruning the decoder changes its internal dynamics. The authors found that you must adapt both the decoder and the projector together. If you only fix one, the performance collapses.
  • Task-Agnostic Pruning: The layers that are redundant for transcribing English (ASR) are the same layers that are redundant for translating French to English (AST). This suggests a global redundancy structure that allows for a single, pruned "backbone" to be used for multiple different speech tasks.

Key Results

  • Efficiency Gains: For Llama 3.1-8B, removing 40% of the decoder layers resulted in a 35% wall-clock speedup and reduced peak GPU memory usage from 15.72 GiB to 10.37 GiB.
  • Performance Retention: 7-8B models maintained high ASR accuracy with only ~60% of their layers remaining.
  • Translation Robustness: In Speech Translation, the researchers could drop 32.1% of layers while keeping BLEU scores (a measure of translation quality) within a strict 10% relative decrease threshold.

Limitations and Future Work

While the results are promising, the authors note a few caveats:

  • The LoRA Factor: While LoRA helps "heal" the model, it also introduces new functional dependencies. Interestingly, LoRA-adapted models were slightly less tolerant of pruning than frozen ones, suggesting that fine-tuning "locks in" certain layers.
  • Task Complexity: The study focused on ASR and AST. It remains to be seen if this level of redundancy holds for more complex "Speech-to-Reasoning" tasks where the LLM's full cognitive capacity might actually be required.
  • Thresholding: The definition of "acceptable" performance loss is subjective. While the authors used a data-driven threshold, different industrial applications might have stricter requirements.

Ultimately, this paper provides a clear recipe for researchers and engineers to stop wasting FLOPs on redundant layers, paving the way for SpeechLLMs that are not just smart, but efficient.

Summary updated

Abstract

Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tasks. Across two LLM families and three scales (1-8B), we show that decoder redundancy is largely inherited from the pretrained LLM: text and speech inputs yield similar redundant blocks. We then measure excess capacity by pruning decoder layers and analysing post-pruning healing to increase robustness. Our findings show that 7-8B models retain good ASR performance with only 60% of decoder layers, and the same trend extends to smaller scales with reduced pruning tolerance. We then generalise to speech translation, and show that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.
BibTeX
@misc{moumen2026measuring,
  author = {Adel Moumen and Guangzhi Sun and Philip C. Woodland},
  title = {{Measuring the Redundancy of Decoder Layers in SpeechLLMs}},
  year = {2026},
  eprint = {2603.05121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05121},
}
Figure