AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions

Chengwei Wei, Yi Ma, Nancy F. ChenarXiv 2026 · arXiv

When an audio language model fails to answer a complex question—like "How fast was that car moving based on its engine sound?"—is it because it couldn't hear the pitch shift, because it doesn't know the Doppler effect formula, or because it just can't do the math?

Current benchmarks can tell us that a model failed, but they rarely tell us why. A new paper, "AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions," introduces a framework to pull back the curtain on these failures.

TL;DR

AuDiR is a diagnostic benchmark that uses controlled "interventions"—such as providing the underlying physics rules or the exact observations as text—to pinpoint whether a model's failure stems from poor perception, missing knowledge, or faulty reasoning. Initial tests on Gemini-3.8-flash reveal that perception is the primary bottleneck: models reason remarkably well once you tell them exactly what they should have heard.

What Problem Does This Paper Address?

Most audio reasoning benchmarks (like MMAR or MMAU) focus on end-to-end accuracy. However, a wrong answer can be caused by three distinct issues:

  1. Missing Knowledge: The model doesn't know the "rule" (e.g., the relationship between frequency and speed).
  2. Failed Perception: The model knows the rule but can't extract the necessary data (the "percept") from the audio clip.
  3. Failed Application: The model has both the rule and the data but fails to combine them correctly.

The authors argue that we need a way to isolate these stages to guide model development effectively.

Methods

The core of AuDiR is a set of 201 items, each containing an audio clip, a question, a task-general Rule (K), and a textual Percept (P) (the observations a human would hear).

To diagnose a model, the researchers run it through four conditions:

  • Condition ① (Base): Audio + Question. Can the model solve it alone?
  • Condition ② (+Knowledge): Audio + Question + Rule. Does knowing the "how-to" fix the error?
  • Condition ③ (+Percept): Audio + Question + Rule + Percept Text. Does knowing the "what-was-heard" fix the error?
  • Condition ④ (No Audio): Question + Rule. A control to ensure the model isn't just guessing from the text.
Fig. 2. AuDiR construction and diagnosis. (a) Candidate percept–target pairings are screened and materialized into four inputs. The audio-removed condition ④ probes text-only answerability; current control results are reported in aggregate. Rejection reasons are detailed in Table VI. (b) The first correct response among conditions ①–③assigns the operational diagnosis; failure under all three is assigned to application. Condition ④is a separate validity control, not a fourth step in this assignment sequence.

By observing which intervention first leads to a correct answer, the authors assign an "operational diagnosis" (e.g., "Failed Perception" if the model only succeeds once the percept text is provided).

Datasets

The AuDiR benchmark consists of 201 items across 18 families (like Doppler speed, heart rate, and melodic intervals) drawn from 9 different sources.

A key innovation here is the Independent Ground Truth (Criterion C4). Many datasets use audio-based labels as ground truth, which is circular—it only tests if the model agrees with another algorithm. AuDiR ensures answers are checked against non-acoustic measurements, such as:

  • VS13: Vehicle speeds recorded via cruise control logs.
  • Sound of Water: Vessel heights measured with a physical ruler.
  • MAFAULDA: Machine rotation speeds measured by a tachometer.
An example item: calculating vehicle speed. The gold answer comes from a cruise-control log, not an audio estimate.

Key Insights and Ideas

  • Perception is the Wall: For the tested model, providing the rule barely helped, but providing the percept text (the "data") caused accuracy to skyrocket. This suggests that the "reasoning" part of "audio reasoning" is actually quite strong in modern LLMs; it's the "audio" part that's lagging.
  • The Circularity Trap: The authors found that many potential data sources had to be rejected because the "ground truth" was actually just an estimate derived from the same audio the model was being asked to analyze.
  • Non-Monotonic Responses: Interestingly, models sometimes get an answer right in the base condition but wrong when given more information. This suggests that some "correct" answers in standard benchmarks might be lucky guesses or based on unintended shortcuts rather than true reasoning.

Key Results

In a preliminary evaluation of Gemini-3.8-flash:

  • Base Accuracy: 29.3%
  • With Rule (+K): 31.3% (A negligible 2% improvement).
  • With Percept Text (+P): 98.5% (A massive jump).
  • No-Audio Control: 23.9% (Close to the 25% chance rate, proving the tasks truly require audio).

The diagnostic breakdown attributed 60.7% of all items to "Failed Perception." Essentially, the model "knew" how to solve the problems but couldn't "hear" the variables needed to plug into the equations.

Limitations and Future Work

  • Verbalizability: AuDiR only works for audio features that can be described in words (like "two pulses per second"). It cannot yet diagnose more abstract qualities like timbre or complex textures.
  • Scale: With only 201 items, the benchmark is a "deep dive" rather than a broad sweep. The authors are currently recording new data (e.g., bouncing balls, pendulums) to expand the set.
  • Human Baseline: The paper lacks a systematic human listening study to confirm that the cues provided in the percept text are indeed audible to humans at the given quality.
  • Model Diversity: The current report focuses on a single model; testing a wider variety of open and closed models is the next logical step.

Summary updated

Abstract

Answer accuracy alone does not distinguish failures in acoustic perception, rule knowledge, and rule application. AuDiR (Audio Diagnostic Reasoning) examines these distinctions through controlled input interventions. Each item contains an audio clip, a multiple-choice question, a task-general rule, and a textual description of the acoustic observations needed to answer. Three conditions progressively supply the rule and observations; a fourth removes the audio while retaining the question and rule. The first correct diagnostic condition assigns an operational response label, without identifying a unique internal cause. The current benchmark contains 201 items in 18 families drawn from 9 sources, with answer references documented separately from percept extraction. In a preliminary evaluation of Gemini-3.8-flash, accuracy is 29.3% without supplied information, 31.3% with the rule, and 98.5% with the observations also supplied. No-audio accuracy is 23.9%, with a confidence interval containing the 25.0% chance rate. A family-level knowledge probe associates incomplete rule recall with larger gains from rule provision. These results show the benefit of supplying acoustic observations as text, while incomplete validation, possible recognition shortcuts, and non-monotonic responses limit stronger failure-attribution claims.
BibTeX
@misc{wei2026audir,
  author = {Chengwei Wei and Yi Ma and Nancy F. Chen},
  title = {{AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions}},
  year = {2026},
  eprint = {2611.12478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12478},
}
Figure