Abstract
Automated audio captioning (AAC) aims to generate descriptions based on audio input, attracting exploration of emerging audio language models (ALMs). However, current evaluation metrics only provide a single score to assess the overall quality of captions without characterizing the nuanced difference by systematically going through an evaluation checklist. To this end, we propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE identifies four main factors that constitute the majority of audio features, specifically sound event, source, attribute and relation. To assess a given caption from an ALM, it is firstly transformed into an audio graph, where each node denotes an entity in the caption and corresponds to a factor. On the one hand, graph matching is conducted from part to whole for a holistic assessment. On the other hand, the nodes contained within each factor are aggregated to measure the factor-level performance. The pros and cons of an ALM can be explicitly and clearly demonstrated through X-ACE, pointing out the direction for further improvements. Experiments show that X-ACE exhibits better correlation with human perception and can detect mismatches sensitively.
Introduction
Evaluating the descriptive accuracy of Audio Language Models (ALMs) is a complex challenge. Traditional metrics like BLEU, ROUGE, or Sentence-BERT often function as "black boxes," providing a single numerical score that fails to diagnose specific model failures. If a model receives a low score, it remains unclear whether it hallucinated a sound, misidentified a source, or failed to capture the temporal sequence of events. X-ACE addresses this by moving beyond single-score metrics to provide an explainable, factor-based assessment paradigm that acts as a systematic checklist for the "who, what, how, and when" of audio descriptions.
Methodology: The X-ACE Paradigm
The X-ACE framework decomposes the evaluation process into four key dimensions: Sound Event (the action), Source (the entity), Attribute (characteristics like volume or distance), and Relation (temporal order).
The pipeline consists of three primary stages:
- Audio Graph Construction: An LLM-based "Factor Extractor" transforms text captions into structured graphs where nodes represent entities and edges represent relations.
- Greedy Graph Matching: The system performs a nuanced alignment between generated and reference graphs, calculating precision and recall for each factor rather than relying on simple binary matching.
- Cross-modal Integration: To ensure audio-text alignment, the framework incorporates a similarity stream using pre-trained HT-SAT and BERT encoders.
AudioCaps-F: A Fine-Grained Benchmark
To validate X-ACE, the researchers developed AudioCaps-F, an enhanced version of the AudioCaps dataset featuring expert-level, fine-grained annotations.
AudioCaps-F includes over 2,100 sound events and nearly 10,000 attributes categorized into nine sub-types, such as pitch and location. This high-resolution data serves as the gold standard for testing how well metrics align with detailed human perception.
Key Insights and Results
X-ACE demonstrates a significantly higher correlation with human judgment (81.8%) compared to traditional metrics like CIDEr (70.8%) or SPICE (59.6%). A critical finding enabled by X-ACE is the "Attribute Vulnerability" in current ALMs; models like Salmonn and Qwen-Audio often identify events correctly but fail to describe how they sound, suffering from low recall in descriptive details. Furthermore, X-ACE's "Relation" factor effectively penalizes models that swap the order of events, a common failure mode that "bag-of-words" metrics overlook.
Conclusion and Future Work
While X-ACE provides unprecedented diagnostic utility, its reliance on LLMs for factor extraction introduces higher computational overhead compared to simpler metrics. Future work will focus on expanding fine-grained annotations to more diverse datasets and improving the underlying cross-modal encoders. By pinpointing specific weaknesses in ALMs, X-ACE provides a clear roadmap for the next generation of audio-language modeling.
Summary updated