A Joint Detection-Classification Model for Audio Tagging of Weakly Labelled Data

Qiuqiang Kong, Yong Xu, Wenwu Wang, Mark D. Plumbley

Abstract

Audio tagging aims to assign one or several tags to an audio clip. Most of the datasets are weakly labelled, which means only the tags of the clip are known, without knowing the occurrence time of the tags. The labeling of an audio clip is often based on the audio events in the clip and no event level label is provided to the user. Previous works have used the bag of frames model assume the tags occur all the time, which is not the case in practice. We propose a joint detection-classification (JDC) model to detect and classify the audio clip simultaneously. The JDC model has the ability to attend to informative and ignore uninformative sounds. Then only informative regions are used for classification. Experimental results on the "CHiME Home" dataset show that the JDC model reduces the equal error rate (EER) from 19.0% to 16.9%. More interestingly, the audio event detector is trained successfully without needing the event level label.

Introduction

In machine listening, audio tagging involves assigning semantic labels (e.g., "dog barking", "music") to audio recordings. A significant challenge in this field is the prevalence of weakly labelled data. In such datasets, we know a specific sound event occurs within a clip, but we lack precise timestamps for its onset and offset.

Traditional approaches, such as the "bag of frames" or "bag of blocks" models, often operate under the assumption that if a tag is present for a clip, it must be present throughout the entire duration. This "always-on" assumption is problematic for short, bursty events like a door slam or a gunshot. Forcing a model to learn features for these events in silent or irrelevant regions introduces noise, hindering generalization and increasing error rates.

Methods: Joint Detection-Classification (JDC)

To address these limitations, we propose the Joint Detection-Classification (JDC) model. This architecture mimics human auditory attention by learning to focus on informative segments while ignoring background noise and silence.

The JDC model processes an audio clip by breaking it into overlapping blocks. Each block is passed through two parallel neural network components:

  1. The Classifier ($y$): Estimates the probability of a tag being present in the current block.
  2. The Detector ($w$): Functions as an attention mechanism, calculating the "informativeness" of the block for a specific tag.
Comparison of model architectures: (a) Bag of Blocks, (b) Global NN, and (c) the proposed Joint Detection-Classification (JDC) model.

The final clip-level prediction is a weighted sum of the block-level classification outputs, where the weights are provided by the detector. If the detector identifies a block as uninformative (e.g., silence), it assigns a near-zero weight, effectively filtering out noise. The entire system is trained end-to-end using only clip-level labels, allowing the model to simultaneously learn "what" the sound is and "where" it occurs.

Experimental Setup and Results

The model was evaluated on the CHiME Home dataset, which contains 4-second domestic audio recordings categorized into seven classes including child speech, percussive sounds, and broadband noise.

Quantitative Performance

The JDC model demonstrated superior performance compared to traditional baselines using the Equal Error Rate (EER) metric:

  • Baseline (MFCC + GMM): 20.9% EER
  • Bag of Blocks (BOB): 19.0% EER
  • JDC Model: 16.9% EER

While the JDC model showed slightly higher error on training data compared to the Bag of Blocks baseline, its significantly better performance on the evaluation set indicates superior generalization. By ignoring uninformative regions, the model avoids overfitting to clip-specific background noise.

Emergent Event Detection

A key finding is that the JDC model learns to perform event detection as a byproduct of the tagging task. Despite never seeing timestamped labels during training, the internal detector weights align closely with manual human annotations.

Visualization of detector activations across different sound classes compared to ground truth annotations.

As shown in the visualization, the detector successfully highlights the specific frames where events like "child speech" (c) or "percussion" (p) occur, matching the ground truth (f).

Conclusion

The JDC model offers a robust framework for audio tagging with weakly labelled data. By integrating an attention-based detector, the model effectively filters out noise and focuses on salient audio events. Future work will explore more complex architectures like CNNs or LSTMs and investigate training stability through techniques like differential learning rates for the detector and classifier components.

Summary updated

Abstract

Audio tagging aims to assign one or several tags to an audio clip. Most of the datasets are weakly labelled, which means only the tags of the clip are known, without knowing the occurrence time of the tags. The labeling of an audio clip is often based on the audio events in the clip and no event level label is provided to the user. Previous works have used the bag of frames model assume the tags occur all the time, which is not the case in practice. We propose a joint detection-classification (JDC) model to detect and classify the audio clip simultaneously. The JDC model has the ability to attend to informative and ignore uninformative sounds. Then only informative regions are used for classification. Experimental results on the "CHiME Home" dataset show that the JDC model reduces the equal error rate (EER) from 19.0% to 16.9%. More interestingly, the audio event detector is trained successfully without needing the event level label.
BibTeX
@inproceedings{kong2017joint,
  author = {Qiuqiang Kong and Yong Xu and Wenwu Wang and Mark D. Plumbley},
  title = {{A Joint Detection-Classification Model for Audio Tagging of Weakly Labelled Data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2017.7952234},
}
Figure