Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition

Xikun Lu (陆锡坤), Yujian Ma, Jinqiu Sang, Ruizhe LiarXiv 2025 · DOI · arXiv

Abstract

This study investigates the internal mechanisms of Low-Rank Adaptation (LoRA) when applied to the Whisper speech model for Speech Emotion Recognition (SER). By employing mechanistic interpretability tools, the researchers identify a "delayed specialization" strategy where task-specific adaptations are concentrated in the final layers. The analysis also reveals functional asymmetries in LoRA matrices and the emergence of subtractive signaling to suppress irrelevant features, providing a first look at how LoRA reshapes the internal logic of speech models.

Introduction

Large pre-trained models like OpenAI’s Whisper are typically optimized for Automatic Speech Recognition (ASR), focusing on lexical content. Adapting these models for Speech Emotion Recognition (SER)—which relies on prosody and tone—presents a significant shift in objective. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have become standard for such adaptations, yet they are often treated as "black boxes." This study aims to uncover how LoRA reshapes Whisper's internal representations to resolve the conflict between its original "hearing" (ASR) and its new "feeling" (SER).

Methodology

The researchers analyzed a LoRA-adapted Whisper-large-v2 encoder using several mechanistic interpretability (MI) techniques:

  • Layer Contribution Probing: Measuring the impact of Attention and MLP sublayers on the residual stream.
  • Logit-Lens Inspection: Projecting intermediate layer outputs through the final classifier to observe internal prediction convergence.
  • Representational Analysis: Using Singular Value Decomposition (SVD) for weight complexity and Centered Kernel Alignment (CKA) for representational similarity.
  • t-SNE Visualization: Mapping high-dimensional states to 2D to observe emotional clustering across depths.

The study utilized the IEMOCAP dataset, focusing on four emotions: anger, happiness, neutrality, and sadness, using a speaker-independent 10-fold cross-validation setup.

Mechanistic Insights

Delayed Specialization and Late-Stage Convergence

A key finding is the "delayed specialization" strategy. In the original frozen Whisper model, internal guesses about emotion are unstable. LoRA maintains general speech features in early layers and defers emotional decision-making to the final 15% of the network (layer 27 and above).

Figure 2: Logit-lens analysis showing KL divergence and prediction overlap (a) alongside t-SNE visualizations of emotional clustering across layers (b).

As shown in Figure 2, the KL divergence between intermediate and final predictions drops sharply in the final layers for the LoRA-adapted model, while the prediction overlap increases. The t-SNE plots confirm that distinct emotional clusters only emerge clearly in the deepest layers.

Subtractive Signaling and Matrix Asymmetry

The analysis revealed that LoRA introduces "counter-directional" signals in later layers. This "subtractive signaling" acts as a filter, actively suppressing ASR-related features that might distract from emotional cues.

Figure 1: Relative contribution of sublayers (a) and cosine similarity between original and updated representations (b).

In Figure 1(b), the negative cosine similarity in the final layers indicates that LoRA's updates are often opposing the direction of the original frozen weights, effectively pruning irrelevant information to refine the emotional signal. Additionally, the study found a functional split between LoRA’s $A$ and $B$ matrices: Matrix $A$ serves as a compression branch, while Matrix $B$ acts as a reconstruction branch.

Performance and Scaling

LoRA-adapted Whisper-large-v2 achieved an Unweighted Recall (UAR) of 0.774, a substantial improvement over the 0.582 UAR of the frozen-encoder baseline. Unlike the frozen model, which exhibited irregular performance across different sizes, LoRA-adapted models showed consistent scaling, with performance improving predictably as model size increased. Interestingly, high-energy emotions like "happiness" were found to require higher LoRA ranks, suggesting they demand more representational capacity than neutral or sad tones.

Conclusion

This work provides a blueprint for understanding how LoRA adapts speech models. The discovery of delayed specialization suggests that "depth-aware" LoRA—targeting only critical late-stage layers—could further optimize computational efficiency. Future research will investigate if these patterns hold across other architectures and tasks.

Summary updated

BibTeX
@misc{2025behind,
  author = {Xikun Lu (陆锡坤) and Yujian Ma and Jinqiu Sang and Ruizhe Li},
  title = {{Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition}},
  year = {2025},
  eprint = {2509.08454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.08454},
}
Figure