One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models

Wenbin Wang, Yuheng Li, Koichiro YoshinoarXiv 2026 · arXiv

TL;DR

SAC-Bench is a massive new benchmark designed to test whether Audio-Language Models (ALMs) actually "listen" to sounds or just follow spoken instructions within an audio clip. The researchers found that even the most capable models frequently ignore acoustic evidence in favor of misleading spoken claims, revealing a significant "linguistic shortcut" bias that persists across semantic, temporal, and spatial tasks.

What Problem Does This Paper Address?

Modern Audio-Language Models are expected to understand both speech and environmental sounds. Usually, these two sources of information agree: if you hear a person say "listen to this dog" followed by a bark, the model succeeds. But does the model succeed because it recognized the bark, or because it simply followed the spoken text?

This paper addresses intra-audio speech–acoustic conflict. This occurs when a single audio waveform contains a spoken claim that contradicts the actual acoustic evidence (e.g., a voice says "the clock is ringing" while a siren is actually sounding). Prior benchmarks mostly looked at contradictions between separate files (like a text prompt vs. an audio file). SAC-Bench probes whether models can remain "faithful" to the raw acoustic facts when a voice inside the same recording is lying to them.

Figure 1: Overview of SAC-Bench. A single audio waveform contains spoken claims and environmental acoustic evidence that may imply conflicting interpretations. SAC-Bench evaluates this conflict across four task groups: event presence, event identity, temporal structure, and spatial configuration.

Methods

The authors developed a rigorous pipeline to create SAC-Bench, which includes over 616,000 unique audio clips.

  1. Acoustic Selection: They curated high-quality environmental recordings from the FSD50K dataset.
  2. Claim Synthesis: They used GPT-4o to generate spoken claims and F5-TTS to synthesize them into natural-sounding speech.
  3. Conflict Generation: They paired sounds with "Compatible" claims (truthful) and "Conflicting" claims (lies).
  4. Spatial Rendering: For spatial tasks, they used advanced audio engineering (Ambisonics and HRTFs) to place the sound source and the speaker in specific 3D locations, allowing them to test if the physical location of a voice interferes with a model's ability to localize a sound.
  5. Evaluation: They tested five state-of-the-art open-weight ALMs (like Qwen3-Omni and Step-Audio-2-mini) using different prompting strategies: neutral, acoustic-grounded (telling the model to focus on sounds), and conflict-aware (warning the model that the speaker might lie).
Figure 2: SAC-Bench construction pipeline, from environmental audio selection and spoken claim synthesis to conflict generation, audio rendering, and evaluation.

Datasets

The primary contribution is SAC-Bench, which dwarfs previous conflict-based benchmarks in scale and scope:

  • 616,031 unique clips (approx. 2,070 hours of audio).
  • 188 sound-event classes across 7 task groups.
  • Tasks include: Event Presence, Event Identity, Temporal State (before/after), Temporal Order, Static Location, Spherical Motion, and Radial Motion.
  • It builds upon FSD50K and AudioSet for its base environmental sounds.
Table 1: Comparison of audio-language benchmarks in speech–acoustic conflict coverage and dataset scale.

Key Insights and Ideas

  • Acoustic Competence $\neq$ Robustness: A model can be nearly perfect at identifying a sound in isolation (audio-only) but fail completely when a voice provides a conflicting label. This suggests models rely on "linguistic shortcuts."
  • The "Recency" Myth: In temporal tasks, models failed even when the misleading speech came after the entire sequence of sounds. This proves the failure isn't just due to the speech physically "masking" or drowning out the sound; it's a high-level reasoning failure.
  • Spatial Interference is Two-Fold: In 3D audio tasks, models were distracted both by what the speaker said ("the bird is moving left") and where the speaker's voice was physically coming from.
  • Prompting is a Double-Edged Sword: Telling a model "the speaker might be lying" can help it recover the truth, but it can also backfire, causing the model to become overly skeptical and perform worse on simple tasks.

Key Results

  • Massive Accuracy Drops: On the "Event Identity" task, model accuracy was 24.8 to 46.5 percentage points lower in conflicting conditions compared to compatible ones.
  • High Speech-Following Rate (SFR): When faced with a conflict, models chose the "lie" told by the speaker 47% to 80% of the time.
  • Temporal Failures: For "Temporal Order," Step-Audio-2-mini dropped from 72.6% accuracy (audio-only) to a staggering 7.5% under conflict.
  • Intervention Effects: Conflict-aware prompting helped Qwen3-Omni improve its Identity accuracy from 39.8% to 59.3%, showing that explicit instructions can partially mitigate these shortcuts.

Limitations and Future Work

  • Synthetic Nature: While the environmental sounds are real, the spoken claims are synthesized and the 3D environments are rendered. Real-world audio with overlapping spontaneous speech might present even tougher challenges.
  • Model "Black Box": The study identifies that models fail, but doesn't fully explain the internal neural mechanisms causing the linguistic bias.
  • Spatial Scope: The spatial analysis was primarily limited to one model (Spatial-Omni) due to the specialized input requirements (4-channel Ambisonics) of such tasks.

The authors suggest that future ALM training should include "conflict-aware" data to force models to ground their reasoning in raw acoustics rather than just following the easiest linguistic path.

Summary updated

Abstract

Audio-language models (ALMs) are evaluated on their understanding of both speech and environmental sounds. What remains underexplored is how they respond when spoken claims contradict acoustic evidence within the same recording. We introduce SAC-Bench, a benchmark of intra-audio speech--acoustic conflict comprising 616,031 unique clips from 188 sound-event classes across seven semantic, temporal, and spatial tasks. It pairs environmental recordings with compatible or contradictory synthesized spoken claims and includes audio-only controls where applicable. Across five open-weight ALMs evaluated on semantic and temporal tasks, high audio-only accuracy often coexists with large losses under conflicting claims; Event Identity accuracy is 24.8--46.5 percentage points lower in conflict than compatible conditions. Temporal Order failures persist even when speech follows the complete event sequence, so simultaneous speech--event masking cannot fully explain them. A spatial case study with Spatial-Omni finds lower motion accuracy associated with both spoken claims and the speech source's physical trajectory. Prompt interventions sometimes restore acoustic accuracy, but can also reduce claim-aligned answers without improving grounding or worsen performance. SAC-Bench thus separates acoustic competence, conflict robustness, and intervention effects.
BibTeX
@misc{wang2026one,
  author = {Wenbin Wang and Yuheng Li and Koichiro Yoshino},
  title = {{One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models}},
  year = {2026},
  eprint = {2611.00325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00325},
}
Figure