TL;DR
Large Audio-Language Models (LALMs) excel at identifying sounds but struggle with temporal reasoning, such as tracking when a sound starts or how long it lasts. This paper introduces a targeted diagnostic benchmark and uses mechanistic interpretability to show that these failures aren't just due to models ignoring audio in favor of text, but rather how they distribute attention across audio tokens. By strategically scaling attention at specific bottleneck layers during inference, the authors improve temporal reasoning accuracy without any fine-tuning or extra training data.
What Problem Does This Paper Address?
While state-of-the-art LALMs are highly capable of acoustic event detection and audio captioning, they frequently stumble when asked to localize events in time or reason about their temporal relationships (e.g., determining which sound happened first).
Prior literature often blames this on "modality imbalance"—the tendency of multimodal models to over-rely on text cues and ignore the audio channel. However, these conclusions have primarily been drawn from behavioral analysis (observing inputs and outputs), which shows correlation but not causation. This paper aims to uncover the actual causal mechanisms behind temporal reasoning failures in LALMs using mechanistic interpretability.
Methods
The authors approach the problem using two distinct lenses:
- Behavioral Analysis: They evaluate models across three input configurations—Audio-only QA (AQA), Caption-only QA (CQA), and Audio-Caption QA (ACQA)—alongside a silence ablation test to verify how much the models truly rely on the audio signal.
- Causal Attention Interventions: To move beyond correlation, they apply training-free interventions directly to the models' attention mechanisms during inference. They compare two techniques:
- Attention Upweighting: Linearly increasing the total attention mass allocated to audio tokens.
- Attention Scaling (ScalingVis): Multiplicatively scaling the pre-softmax attention logits by a coefficient ($\alpha$). This either sharpens the attention distribution ($\alpha > 1$) to focus on specific tokens or smoothes it ($\alpha < 1$) to distribute focus more evenly.
They also experiment with targeting different token positions: the final prompt token, task-specific keywords (e.g., "earliest", "longest"), or a combination of both.
Datasets
The authors introduce a new diagnostic benchmark specifically designed for mechanistic analysis. It consists of 1,657 multiple-choice questions derived from the TACOS dataset (which contains real-world audio clips from Freesound with precise temporal annotations).
The benchmark isolates three foundational temporal tasks:
- Earliest Onset (EO): Identify which of four sound events started first.
- Latest Offset (LO): Identify which of four sound events ended last.
- Longest Duration (LD): Identify which of four sound events lasted the longest.
To ensure the tasks strictly require temporal reasoning, correct answers are separated from distractors by at least one second.
Key Insights and Ideas
- It’s Not How Much You Listen, It’s How You Listen: Simply boosting total attention to audio (upweighting) is relatively ineffective. Instead, redistributing attention across audio tokens (scaling) yields significantly higher error correction rates. This disproves the simple narrative that modality imbalance alone causes these failures.
- Failures are Architecture-Dependent: Different models fail for different internal reasons. Audio-Flamingo-3 benefits from attention sharpening ($\alpha = 2.0$), meaning its internal attention is directed toward the right features but lacks precision. Conversely, DeSTA-2.5-Audio benefits from smoothing ($\alpha = 0.2$), indicating its attention is fundamentally misaligned and needs to be spread out.
- Keywords Act as Attention Anchors: Interventions that target both the final prompt token and task-relevant keywords (like "earliest") perform the best. This implies that the linguistic framing of a temporal question heavily dictates how effectively a model parses the accompanying audio.
Key Results
- Audio Dependency Verified: During silence ablation (replacing audio with silence), model performance dropped to near-chance (~25%), confirming that the benchmark cannot be solved using textual shortcuts alone.
- High Error Correction (Fix Rates): On samples where the models initially guessed incorrectly, attention scaling fixed up to 24.3% of errors for Audio-Flamingo-3 and 21.2% for DeSTA.
- Inference-Time Performance Boost: By applying layer-targeted attention scaling to identified bottleneck layers (Layer 20 for Audio-Flamingo-3 and Layer 9 for DeSTA), the authors achieved a training-free accuracy boost, raising the average baseline accuracy from 55.9% to 59.1%.
| Intervention Type (Kwd+Last) | Audio-Flamingo-3 (Avg Fix Rate) | DeSTA-2.5-Audio (Avg Fix Rate) |
|---|---|---|
| Attention Upweighting | 15.4% ($\alpha=0.5$) | 9.4% ($\alpha=0.5$) |
| Attention Scaling | 20.5% ($\alpha=2.0$) | 20.1% ($\alpha=0.2$) |
Limitations and Future Work
The authors note that while attention distribution is a major factor, it is not the only cause of failure; their interventions do not achieve a 100% fix rate. Weak representations from the underlying audio encoder could still be bottlenecking performance upstream.
Additionally, this work focuses on a training-free, inference-time patch. The authors suggest that future work should leverage these insights to design better training-time loss functions or architectures that inherently foster well-distributed cross-modal attention.
Summary updated