Introduction
As Large Language Models (LLMs) evolve into Large Audio-Language Models (LALMs), the AI community has seen a surge in benchmarks without a unified framework to measure progress. This survey addresses the "multimodal explosion" where models like GPT-4o and Gemini process audio signals directly. However, evaluation methods remain fragmented. Researchers often struggle to distinguish whether a model truly understands an audio signal or is merely leveraging its internal text-based knowledge. Without a structured taxonomy, it is nearly impossible to identify which specific capabilities—like real-time turn-taking or detecting subtle sarcasm—are still missing.
A Unified Taxonomy for LALM Evaluation
To address these challenges, we propose a systematic taxonomy that organizes LALM evaluation into four distinct pillars:
- General Auditory Awareness and Processing: Evaluates the model's "ears"—its ability to identify speakers, detect emotions, and transcribe speech accurately.
- Knowledge and Reasoning: Evaluates the model's "brain"—its capacity to apply world knowledge to audio contexts, such as diagnosing a medical condition from a cough.
- Dialogue-oriented Ability: Evaluates "social skills," including full-duplex capabilities like handling interruptions and backchanneling.
- Fairness, Safety, and Trustworthiness: Evaluates the "ethics" of the model, focusing on biases, audio-based jailbreaking, and acoustic hallucinations.
The detailed structure of this taxonomy, mapped to specific research sections and existing benchmarks, provides a roadmap for holistic model assessment.
The Benchmarking Landscape
The survey categorizes dozens of existing benchmarks rather than introducing a single new one. Key highlights include:
- Dynamic-SUPERB: A collaborative suite with over 180 tasks for general processing.
- SALMon: Designed for nuanced auditory perception, such as detecting tone-content mismatches.
- VoiceBench & VoxEval: Spoken adaptations of popular text benchmarks (e.g., MMLU) to test world knowledge.
- Full-Duplex-Bench: A framework for real-time conversational dynamics.
- Foundational Corpora: Datasets like LibriSpeech (ASR), VoxCeleb (Speaker ID), and AudioCaps (Captioning) remain the building blocks for modern benchmarks.
Critical Findings and Insights
Our analysis reveals several key trends and vulnerabilities in current LALMs:
- Instruction Following Degradation: Adapting LLMs to audio often leads to "catastrophic forgetting" of instruction-following capabilities. Models may process audio correctly but fail to adhere to complex output formatting constraints.
- End-to-End (E2E) vs. Cascaded Systems: While cascaded systems (ASR + text LLM) often excel at knowledge tasks, E2E models are significantly better at capturing paralinguistic cues like sarcasm and emotion that ASR typically strips away.
- Audio-Specific Safety Risks: Models safe in text can be "jailbroken" via audio. Malicious prompts delivered with specific accents or background noise often bypass safety filters designed for text.
- The Hallucination Gap: LALMs frequently suffer from "object hallucination," falsely identifying sounds based on statistical co-occurrence in training data rather than actual acoustic evidence.
Performance Gaps and Future Directions
Current LALMs still lag behind human-level perception, particularly in tasks where acoustic signals contradict semantic content. Furthermore, data contamination is a growing concern, as many benchmarks rely on public datasets that models may have encountered during training.
Future research should focus on:
- Linguistic and Cultural Diversity: Expanding benchmarks beyond English and Mandarin to include low-resource languages.
- Auditory-Specific Safety: Moving beyond "bad words" to evaluate "auditory comfort" (e.g., avoiding harsh tones or disrespectful vocal manners).
- Personalization: Evaluating adaptation to specific user voices, preferences, or speech disorders.
- Unified Metrics: Developing balanced measures for helpfulness and harmlessness in voice interactions.
Summary updated