Abstract
Large language models (LLMs) have advanced in text and vision, but their reasoning on audio remains limited. Most existing methods rely on dense audio embeddings, which are difficult to interpret and often fail on structured reasoning tasks. Caption-based approaches, introduced in recent benchmarks such as MMAU, improve performance by translating audio into text, yet still depend on dense embeddings as input, offering little insight when models fail. We present SAR-LM, a symbolic audio reasoning pipeline that builds on this caption-based paradigm by converting audio into structured, human-readable features across speech, sound events, and music. These symbolic inputs support both reasoning and transparent error analysis, enabling us to trace failures to specific features. Across three benchmarks, MMAU, MMAR, and OmniBench, SAR-LM achieves competitive results, while prioritizing interpretability as its primary contribution.
Introduction
While Large Language Models (LLMs) have become remarkably adept at reasoning through text and images, audio remains a challenging frontier. Most current systems treat audio as a "black box" of dense embeddings—mathematical vectors that are efficient for machines but indecipherable to humans. When these models fail to answer complex questions about a soundscape, diagnosing the root cause is nearly impossible.
SAR-LM (Symbolic Audio Reasoning with Large Language Models) addresses this by swapping opaque embeddings for human-readable "symbols." This framework converts raw audio into structured text—such as transcripts, musical chords, and sound event tags—before reasoning over them with an LLM, creating a "glass box" view of the AI's logic.
The SAR-LM Pipeline
The SAR-LM pipeline consists of four distinct stages designed to maximize both performance and transparency:
- Symbolic Feature Extraction: Instead of a single end-to-end model, SAR-LM employs a suite of specialized "expert" models. Speech is handled by Whisper-large (transcripts) and DAWN Transformer (emotion); music is processed via MT3 (notes), Chordino (chords), and Musicnn (tags); and environmental sounds are detected and timestamped using PANNs.
- Prompt Construction: Extracted features are formatted into a structured text prompt. A content-aware filter ensures only relevant data is included (e.g., musical notes are omitted if no music is detected).
- Dynamic Feature Selection: A "GPT-style agent" (utilizing Gemini 2.5 Pro) analyzes the user's question and the available features, selecting only the most relevant symbols to include in the final prompt to reduce "prompt noise."
- LLM Reasoning: The final summarized prompt is processed by a high-performing LLM (such as Gemini 2.5 Pro or Qwen3) to generate the answer.
Evaluation and Results
SAR-LM was evaluated across three diverse benchmarks: MMAU (Massive Multi-task Audio Understanding), MMAR (Massive Multi-task Audio Reasoning), and OmniBench (a tri-modal situational awareness benchmark).
As shown in the results, SAR-LM using Gemini 2.5 Pro achieved a top-tier 73.5% accuracy on MMAU and 69.3% on MMAR, outperforming several existing baselines. Notably, on the MMAR benchmark, SAR-LM significantly exceeded the previous best-reported baseline of 65.6%, suggesting that symbolic representations are particularly effective for tasks requiring long-context reasoning.
Interpretability and the "Interpretability Tax"
A key finding of this research is the "interpretability tax." While end-to-end models that process raw waveforms still hold a slight edge in raw accuracy, the symbolic approach is surprisingly competitive, with a performance gap of only about 5% on MMAU.
The primary advantage of SAR-LM is the ability to trace the root cause of failures. In error analyses, researchers could identify when a reasoning failure was actually a detection failure (e.g., a sound event detector missing a quiet sound) rather than a logic error by the LLM. Furthermore, using an LLM agent to filter irrelevant features consistently improved performance, raising overall accuracy from 66.6% to 68.1% in certain configurations.
Conclusion
SAR-LM demonstrates that symbolic audio reasoning is a viable and highly interpretable alternative to dense embedding models. While challenges remain regarding computational costs and error propagation from expert models, the framework provides a necessary step toward transparent audio AI. Future work will focus on streamlining the pipeline with universal sound recognition backends and improving the detection of low-energy acoustic events.
Summary updated