X-ACE: Explainable and Multi-factor Audio Captioning Evaluation

Qian Wang, Jia-Chen Gu, Zhen-Hua LingACL 2024 · OpenReview

Abstract

Automated audio captioning (AAC) aims to generate descriptions based on audio input, attracting exploration of emerging audio language models (ALMs). However, current evaluation metrics only provide a single score to assess the overall quality of captions without characterizing the nuanced difference by systematically going through an evaluation checklist. To this end, we propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE identifies four main factors that constitute the majority of audio features, specifically sound event, source, attribute and relation. To assess a given caption from an ALM, it is firstly transformed into an audio graph, where each node denotes an entity in the caption and corresponds to a factor. On the one hand, graph matching is conducted from part to whole for a holistic assessment. On the other hand, the nodes contained within each factor are aggregated to measure the factor-level performance. The pros and cons of an ALM can be explicitly and clearly demonstrated through X-ACE, pointing out the direction for further improvements. Experiments show that X-ACE exhibits better correlation with human perception and can detect mismatches sensitively.

Introduction

Evaluating the descriptive accuracy of Audio Language Models (ALMs) is a complex challenge. Traditional metrics like BLEU, ROUGE, or Sentence-BERT often function as "black boxes," providing a single numerical score that fails to diagnose specific model failures. If a model receives a low score, it remains unclear whether it hallucinated a sound, misidentified a source, or failed to capture the temporal sequence of events. X-ACE addresses this by moving beyond single-score metrics to provide an explainable, factor-based assessment paradigm that acts as a systematic checklist for the "who, what, how, and when" of audio descriptions.

Methodology: The X-ACE Paradigm

The X-ACE framework decomposes the evaluation process into four key dimensions: Sound Event (the action), Source (the entity), Attribute (characteristics like volume or distance), and Relation (temporal order).

The X-ACE framework pipeline, illustrating Factor Graph Matching and Cross-modal Matching components.

The pipeline consists of three primary stages:

  1. Audio Graph Construction: An LLM-based "Factor Extractor" transforms text captions into structured graphs where nodes represent entities and edges represent relations.
  2. Greedy Graph Matching: The system performs a nuanced alignment between generated and reference graphs, calculating precision and recall for each factor rather than relying on simple binary matching.
  3. Cross-modal Integration: To ensure audio-text alignment, the framework incorporates a similarity stream using pre-trained HT-SAT and BERT encoders.

AudioCaps-F: A Fine-Grained Benchmark

To validate X-ACE, the researchers developed AudioCaps-F, an enhanced version of the AudioCaps dataset featuring expert-level, fine-grained annotations.

Distribution of sound events, sources, and attributes in the AudioCaps-F dataset.

AudioCaps-F includes over 2,100 sound events and nearly 10,000 attributes categorized into nine sub-types, such as pitch and location. This high-resolution data serves as the gold standard for testing how well metrics align with detailed human perception.

Key Insights and Results

X-ACE demonstrates a significantly higher correlation with human judgment (81.8%) compared to traditional metrics like CIDEr (70.8%) or SPICE (59.6%). A critical finding enabled by X-ACE is the "Attribute Vulnerability" in current ALMs; models like Salmonn and Qwen-Audio often identify events correctly but fail to describe how they sound, suffering from low recall in descriptive details. Furthermore, X-ACE's "Relation" factor effectively penalizes models that swap the order of events, a common failure mode that "bag-of-words" metrics overlook.

Conclusion and Future Work

While X-ACE provides unprecedented diagnostic utility, its reliance on LLMs for factor extraction introduces higher computational overhead compared to simpler metrics. Future work will focus on expanding fine-grained annotations to more diverse datasets and improving the underlying cross-modal encoders. By pinpointing specific weaknesses in ALMs, X-ACE provides a clear roadmap for the next generation of audio-language modeling.

Summary updated

Abstract

Automated audio captioning (AAC) aims to generate descriptions based on audio input, attracting exploration of emerging audio language models (ALMs). However, current evaluation metrics only provide a single score to assess the overall quality of captions without characterizing the nuanced difference by systematically going through an evaluation checklist. To this end, we propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE identifies four main factors that constitute the majority of audio features, specifically sound event, source, attribute and relation. To assess a given caption from an ALM, it is firstly transformed into an audio graph, where each node denotes an entity in the caption and corresponds to a factor. On the one hand, graph matching is conducted from part to whole for a holistic assessment. On the other hand, the nodes contained within each factor are aggregated to measure the factor-level performance. The pros and cons of an ALM can be explicitly and clearly demonstrated through X-ACE, pointing out the direction for further improvements. Experiments show that X-ACE exhibits better correlation with human perception and can detect mismatches sensitively.
BibTeX
@inproceedings{wang2024x,
  author = {Qian Wang and Jia-Chen Gu and Zhen-Hua Ling},
  title = {{X-ACE: Explainable and Multi-factor Audio Captioning Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=KaqeWVzTXg},
}
Figure