Deep Feature Embedding and Hierarchical Classification for Audio Scene Classification

Lam Dang Pham, Ian McLoughlin, Huy Phan, Ramaswamy Palaniappan, Alfred Mertins

TL;DR

This paper introduces a novel approach for Acoustic Scene Classification (ASC) by combining deep feature embedding learning with a hierarchical classification strategy. By first learning robust audio embeddings and then organizing scene categories into a hierarchy, the system achieves significant accuracy improvements over existing baselines.

What Problem Does This Paper Address?

Acoustic Scene Classification (ASC) aims to identify the environment or scene based on its audio characteristics (e.g., distinguishing between a "bus" and a "metro station"). This task is challenging due to the presence of foreground sounds, interfering noise, and a wide variety of spectral and temporal patterns. Existing state-of-the-art methods often rely on complex ensemble models that combine diverse input features and learning architectures. However, these approaches can be computationally expensive and may not fully exploit the inherent structure within the scene categories themselves. This paper addresses the need for a more efficient and structured approach to ASC, aiming to improve classification accuracy by leveraging both powerful feature representations and a more intelligent classification scheme.

Methods

The proposed system has two main components:

  1. Deep Feature Embedding Learning:
    • A deep Convolutional Neural Network (CNN) is trained to learn a feature embedding from audio signals.
  2. Audio is first converted into time-frequency representations like Gammatone spectrograms.
  3. These spectrograms are then processed through a CNN architecture (similar to VGG, detailed in Table I).
  4. Crucially, the network is trained using Mixup data augmentation to generate synthetic training samples by linearly interpolating between existing samples and their labels.
  5. Instead of a standard cross-entropy loss, a Kullback-Leibler (KL) divergence loss is used due to the non-one-hot labels from Mixup.
  6. The final output of the CNN's last convolutional layer is used as a high-level feature vector (256-dimensional embedding).
  7. Hierarchical Classification:
    • The traditional "flat" ASC task (classifying all categories at once) is restructured into a two-level hierarchy.
  8. Similar scene categories are grouped into meta-categories (e.g., "vehicle," "indoor," "outdoor"). This forms the first level of classification.
  9. Classification proceeds top-down: first, the meta-category is predicted, and then, within the predicted meta-category, the specific scene is classified.
  10. Deep Neural Networks (DNNs), referred to as DNN-02s, are used as classifiers at each level.
  11. These DNNs are trained using a combination of KL-divergence loss (for the initial classification) and triplet loss. Triplet loss is employed to explicitly minimize distances between similar samples and maximize distances between dissimilar ones, enhancing discrimination.
  12. The CNN trained for feature embedding is reused as a fixed feature extractor, feeding its embeddings into the DNN-02 classifiers.

The authors also explore an ensemble approach where predictions from three different time-frequency representations (Gammatone, log-Mel, and CQT spectrograms) are combined to further boost performance.

Datasets

The experiments were conducted on the DCASE 2018 Task 1A and 1B development datasets.

  • Task 1A: Recordings from a single device (device A).
  • Task 1B: Reuses Task 1A recordings and adds new recordings from two different devices (devices B and C), introducing a device mismatch challenge. The datasets were split into training and evaluation subsets.

Key Insights and Ideas

  • Hierarchical Structure is Key: Organizing scene categories into a hierarchy significantly improves classification performance. This is because it allows classifiers to focus on distinguishing between acoustically similar categories within a smaller, more relevant group, rather than trying to differentiate all categories simultaneously.
  • Deep Embeddings as a Foundation: Learning robust, high-level feature embeddings with a CNN is crucial. These embeddings capture essential acoustic characteristics that are then effectively utilized by the hierarchical classifiers.
  • Triplet Loss for Discrimination: The triplet loss function proves effective in enhancing the discriminative power of the hierarchical classifiers, leading to better separation of acoustically similar scenes.
  • Mixup for Robustness: Mixup data augmentation helps create more robust models by generating diverse training samples, which is particularly beneficial when dealing with variations in audio signals.
  • Ensembling Time-Frequency Representations: Combining different time-frequency representations (Gammatone, log-Mel, CQT) offers complementary information, leading to improved performance across all tested systems.

Key Results

  • The proposed system achieved significant accuracy gains over the DCASE 2018 baseline: 15.6% absolute on Task 1A and 16.6% absolute on Task 1B.
  • On Task 1A, the proposed system with triplet loss achieved an accuracy of 78.0%, outperforming many state-of-the-art methods.
  • On Task 1B, the system achieved 66.9% accuracy, demonstrating strong performance even with device mismatches.
  • The meta-category classifier achieved an average accuracy of 94%, indicating the effectiveness of the initial hierarchical grouping.
  • Gammatone spectrograms generally performed best among individual time-frequency representations, while CQT performed worst. However, ensembling these representations consistently improved results.

Limitations and Future Work

  • Task 1B Performance Drop with Triplet Loss: The paper notes that using triplet loss on Task 1B led to a decrease in accuracy compared to the system without it. The authors attribute this to potential device mismatch issues or insufficient training data on the target devices (B & C).
  • Future Work: The authors suggest exploring deeper hierarchical schemes with a larger number of categories and investigating data-driven clustering approaches to automatically define the hierarchical structure. Further experimentation with more complex ensemble strategies is also a promising direction.

Summary updated

Abstract

In this work, we propose an approach that features deep feature embedding learning and hierarchical classification with triplet loss function for Acoustic Scene Classification (ASC). In the one hand, a deep convolutional neural network is firstly trained to learn a feature embedding from scene audio signals. Via the trained convolutional neural network, the learned embedding embeds an input into the embedding feature space and transforms it into a high-level feature vector for representation. In the other hand, in order to exploit the structure of the scene categories, the original scene classification problem is structured into a hierarchy where similar categories are grouped into meta-categories. Then, hierarchical classification is accomplished using deep neural network classifiers associated with triplet loss function. Our experiments show that the proposed system achieves good performance on both the DCASE 2018 Task 1A and 1B datasets, resulting in accuracy gains of 15.6% and 16.6% absolute over the DCASE 2018 baseline on Task 1A and 1B, respectively.
BibTeX
@inproceedings{pham2020deep,
  author = {Lam Dang Pham and Ian McLoughlin and Huy Phan and Ramaswamy Palaniappan and Alfred Mertins},
  title = {{Deep Feature Embedding and Hierarchical Classification for Audio Scene Classification}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=cNaqpjx29Q},
}
Figure