TL;DR
While modern Audio Language Models (ALMs) are excellent at transcribing what is said, they often fail to comprehend how it is said. This paper reveals a striking "encoder-output gap": ALMs strongly capture paralinguistic features like tone and emotion in their audio encoders, but text-centric training objectives cause their decoders to discard this information, rendering them effectively blind to acoustic style at the output level.
What Problem Does This Paper Address?
Speech is much more than literal text; it carries rich paralinguistic signals through whispering, laughing, changes in pitch, and pacing. While standalone speech encoders are highly capable of capturing these nuances, modern Audio Language Models (ALMs) connect these encoders to Large Language Models (LLMs) via a "projector" to generate text responses.
Until now, it has been unclear whether the full ALM pipeline preserves this stylistic information or if it gets lost because the models are primarily trained to recover text. The authors investigate a fundamental question: Do these models actually capture and use paralinguistic information, or do they ignore it in favor of spoken content?
Methods
The authors introduce a comprehensive, four-part mechanistic analysis framework to trace paralinguistic information through every stage of four open-source models (Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B):
- Centered Kernel Alignment (CKA): Measures the geometric similarity of internal representations across different speaking styles.
- Linear Probing: Trains simple classifiers on hidden states layer-by-layer using Leave-One-Speaker-Out (LOSO) cross-validation to determine if style information is linearly accessible, even if geometrically altered.
- Open-Ended Tone Prediction: Prompts the full models to describe the tone of speech in natural language, mapping their responses to predefined categories.
- Content-Prosody Leakage Ratio: A novel metric based on Normalized Mutual Information (NMI) that quantifies whether a model's final tone predictions are driven by semantic text content or actual acoustic style.
Datasets
The study evaluates models using the Expresso dataset, which features the same text utterances spoken by the same speakers across 7 distinct, controlled speaking styles (confused, default, enunciated, happy, laughing, sad, whisper). This setup perfectly isolates paralinguistic delivery from semantic text content.
Key Insights and Ideas
- The Encoder-Output Gap: There is a massive disconnect between what an ALM encodes internally and what it outputs. Encoders achieve high accuracy (~82–85%) during internal probing, but final output tone perception accuracy plummets to between 19.7% and 53.7%.
- Projectors Reshape, They Don't Destroy: The projector (the bridge between audio and text spaces) causes a sharp drop in geometric similarity (CKA), but linear probing accuracy remains stable. This proves the projector acts as a geometric transformer rather than an information bottleneck.
- Two Output Behaviors: Models fall into two categories:
- Content-driven: Tone predictions are dictated by text semantics (e.g., predicting "angry" for the text "Go to hell!" even if it was spoken with a laugh or a whisper).
- Acoustic-driven: Predictions genuinely track the acoustic signal.
- Objectives Over Architecture: Qwen2-Audio and Qwen2.5-Omni share nearly identical encoders and LLM backbones, yet Omni is highly acoustic-driven while Qwen2-Audio is almost entirely content-driven. Omni’s speech-generation training objective forces its decoder to keep acoustic features alive, whereas text-only prediction objectives provide no such incentive.
Key Results
- Encoder Peak: All four models peak in style encoding in the late encoder layers (the top third of the encoder) at ~82–85% probing accuracy, well above the 14.3% chance level.
- Qwen2.5-Omni (Acoustic-Driven): Achieved the highest overall tone perception accuracy at 53.70%, dominating on acoustically extreme styles like whispering (93.94%) and laughing (85.11%). It registered a low leakage ratio of 0.272, confirming it listens to the audio.
- Qwen2-Audio (Content-Driven): Suffered from severe acoustic blindness at the output level, with an overall tone accuracy of just 19.68% and a leakage ratio of 0.977 (meaning its predictions were almost entirely driven by text content).
- Chroma-4B: Exhibited a unique two-phase pattern where style information collapsed completely at the backbone entry boundary before partially recovering, finishing with an overall accuracy of 22.57% and a leakage ratio of 0.844.
- The Entanglement Problem: The authors attempted a targeted intervention using a Gradient Reversal Layer (GRL) on Qwen2-Audio’s linear projector to suppress content and maximize style. The intervention failed to change the leakage ratio, proving that style and content are linearly entangled within the encoder space itself and cannot be separated by a simple linear map.
Limitations and Future Work
- Speaker and Style Scale: The analysis relies on four speakers from the Expresso dataset; generalization to larger populations or more subtle paralinguistic traits (like accents or health indicators) remains unproven.
- Temporal Dynamics: The use of uniform mean pooling discards temporal dynamics, which could underrepresent highly dynamic styles like laughing.
- Output Modality: The study only evaluates text outputs. Whether speech-generating models like Omni preserve prosody when generating audio responses remains an open question.
- Future Directions: The findings rule out linear projector-level fixes. Instead, the authors motivate three future directions: utilizing nonlinear projectors, implementing encoder-level contrastive objectives, and introducing auxiliary style-prediction tasks during instruction tuning.
Summary updated