Exploring Differences between Human Perception and Model Inference in Audio Event Recognition

Yizhou Tan, Yanru Wu, Yuanbo Hou, Xin Xu, Hui Bu, Shengchen Li, Dick Botteldooren, Mark D. PlumbleyarXiv 2024 · OpenReview

Abstract

Audio Event Recognition (AER) traditionally focuses on detecting and identifying audio events. Most existing AER models tend to detect all potential events without considering their varying significance across different contexts. This makes the AER results detected by existing models often have a large discrepancy with human auditory perception. Although this is a critical and significant issue, it has not been extensively studied by the Detection and Classification of Sound Scenes and Events (DCASE) community because solving it is time-consuming and labour-intensive. To address this issue, this paper introduces the concept of semantic importance in AER, focusing on exploring the differences between human perception and model inference. This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators. Through labelling frequency and variance, the MAFAR dataset facilitates the quantification of semantic importance and analysis of human perception. By comparing human annotations with the predictions of ensemble pre-trained models, this paper uncovers a significant gap between human perception and model inference in both semantic identification and existence detection of audio events. Experimental results reveal that human perception tends to ignore subtle or trivial events in the event semantic identification, while model inference is easily affected by events with noises. Meanwhile, in event existence detection, models are usually more sensitive than humans.

Introduction

In the field of computer audition, models have reached high proficiency in labeling specific sounds like "car horns" or "bird chirps." However, a fundamental issue remains: most models treat every detectable sound with equal weight. In contrast, human perception is highly contextual and filtered by "semantic importance." For instance, a car engine might be ignored as background noise in a bustling city but recognized as a significant event in a quiet forest.

Current evaluation metrics, such as mean Average Precision (mAP), encourage models to be hyper-sensitive, often leading to "over-detection" of subtle or irrelevant sounds that humans naturally filter out. This paper investigates the discrepancy between model inference (objective calculation of sound presence) and human perception (subjective noticing of events).

Methods and the MAFAR Dataset

To study this gap, the authors developed the Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset. It contains 180 minutes of real-world audio recorded via GoPro in four major cities: Shanghai, London, Gent, and Rotterdam.

Unlike standard datasets with a single ground truth, each clip in MAFAR was independently labeled by 10 professional annotators. Crucially, annotators were not provided with a predefined list of labels; instead, they were instructed to label only "prominent and noticeable" events using their own words. To align these free-form descriptions with the standard 527-class AudioSet ontology, the researchers utilized GPT-4 for semantic mapping.

The study compared these human "foreground" annotations against an ensemble of six state-of-the-art AI models (including EAT and PANNs). Disagreement scores were calculated for both humans ($A_h$) and models ($A_m$) to identify which audio classes caused the most confusion for each group.

Key Results

Semantic Identification Discrepancy

The analysis revealed that humans and models struggle with different types of audio events. Humans show high consistency in ignoring "trivial" sounds like paper tearing or coughing, which they perceive as background. In contrast, models are easily distracted by these subtle events, leading to high disagreement between AI and human labels for such classes. Conversely, models struggle significantly with noisy or overlapping events, such as sirens or heavy traffic, where the AI's pattern recognition is easily confounded.

Figure 3: Disagreement analysis across sound classes for humans and models.

Existence Detection and Model Sensitivity

In terms of "existence detection"—simply determining if a sound is present—models proved to be far more sensitive than humans. The study found that model performance is highly dependent on the "human agreement threshold." As the requirement for human consensus increases (i.e., only considering sounds noticed by a majority of the 10 annotators), the model's precision and F-measure scores decline significantly. This indicates that models are frequently detecting sounds that humans do not consider semantically important.

Figure 4: Model performance scores relative to the threshold of human agreement.

Conclusion

The findings suggest that current AER research may be limited by optimizing for "perfect microphone" performance rather than "intelligent listener" relevance. While models are technically sensitive, they lack the selective filtering inherent in human perception. Future work will focus on expanding the MAFAR dataset and developing new training objectives and metrics that incorporate semantic importance, moving toward more human-centric audio recognition systems.

Summary updated

Abstract

Audio Event Recognition (AER) traditionally focuses on detecting and identifying audio events. Most existing AER models tend to detect all potential events without considering their varying significance across different contexts. This makes the AER results detected by existing models often have a large discrepancy with human auditory perception. Although this is a critical and significant issue, it has not been extensively studied by the Detection and Classification of Sound Scenes and Events (DCASE) community because solving it is time-consuming and labour-intensive. To address this issue, this paper introduces the concept of semantic importance in AER, focusing on exploring the differences between human perception and model inference. This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators. Through labelling frequency and variance, the MAFAR dataset facilitates the quantification of semantic importance and analysis of human perception. By comparing human annotations with the predictions of ensemble pre-trained models, this paper uncovers a significant gap between human perception and model inference in both semantic identification and existence detection of audio events. Experimental results reveal that human perception tends to ignore subtle or trivial events in the event semantic identification, while model inference is easily affected by events with noises. Meanwhile, in event existence detection, models are usually more sensitive than humans.
BibTeX
@misc{tan2024exploring,
  author = {Yizhou Tan and Yanru Wu and Yuanbo Hou and Xin Xu and Hui Bu and Shengchen Li and Dick Botteldooren and Mark D. Plumbley},
  title = {{Exploring Differences between Human Perception and Model Inference in Audio Event Recognition}},
  year = {2024},
  eprint = {2409.06580},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.06580},
}
Figure