Multi-view Audio and Music Classification

Huy Phan, Huy Le Nguyen, Oliver Y. Chén, Lam Dang Pham, Philipp Koch, Ian McLoughlin, Alfred Mertins

TL;DR

This paper introduces a novel multi-view learning approach for audio and music classification. By adaptively blending the learning signals from different audio feature representations (views), their method effectively harmonizes the training of individual subnetworks, leading to superior classification performance compared to standard concatenation and late-fusion techniques.

What Problem Does This Paper Address?

Audio and music classification tasks often benefit from using multiple types of low-level audio features, such as Mel-scale spectrograms, Gammatone spectrograms, CQT spectrograms, and raw waveforms. These different features can be thought of as "views" of the same underlying audio data, and each view might capture complementary information. While combining these views (multi-view learning) is intuitively appealing, naive fusion methods like simple concatenation or late fusion can lead to suboptimal performance. This is because individual subnetworks processing each view learn at different rates and may overfit at different times, causing the fused representation to be less effective than the best single view. The challenge is to effectively combine these diverse views in a way that leverages their strengths without being hindered by their differing learning dynamics.

Methods

The authors propose a multi-view deep learning architecture consisting of four subnetworks, each dedicated to processing one of the four chosen low-level audio representations (Mel-scale spectrogram, Gammatone spectrogram, CQT spectrogram, and raw waveform). These subnetworks are implemented as Convolutional Recurrent Neural Networks (CRNNs).

The core innovation lies in their adaptive gradient blending mechanism. Instead of simply concatenating the embeddings from each subnetwork for a final classification, their network also maintains individual classification branches for each subnetwork. These individual branches allow the system to monitor the learning behavior (generalization and overfitting) of each view-specific subnetwork.

A novel weighting scheme is then employed to adaptively adjust the contribution of each subnetwork's loss to the overall training objective. This is achieved by calculating weights based on a generalization measure ($G^{(k)}$) and an overfitting measure ($O^{(k)}$). Subnetworks that are generalizing well receive higher weights, encouraging their learning, while those that are overfitting receive lower weights, slowing them down. This adaptive weighting ensures that the views contribute proportionally to the final multi-view embedding based on their current learning status, preventing the issue of "out-of-sync" learning dynamics.

Additionally, the paper proposes a self-ensemble technique where the outputs of the individual view-specific classifiers and the multi-view classifier are combined using the learned weights to produce a final prediction.

Datasets

The proposed method was evaluated on three standard audio and music classification datasets:

  • ESC-50: For environmental sound classification.
  • DCASE2016 Task 1: For audio scene classification.
  • GTZAN: For music genre classification.

The authors used standard cross-validation protocols for each dataset. No new benchmarks were introduced.

Key Insights and Ideas

  • Learning Dynamics Matter: The paper strongly emphasizes that simply combining features (concatenation) or outputs (late fusion) from independently trained models is insufficient. The differing learning speeds and overfitting patterns of individual view-processing subnetworks are critical factors that need to be managed.
  • Monitoring Individual Learning: Having separate classification branches for each view is a clever way to gain insight into their individual learning progress, specifically their generalization and overfitting tendencies.
  • Adaptive Weighting for Harmonious Learning: The adaptive gradient blending mechanism is the key to harmonizing the learning rhythms of different views. By dynamically adjusting weights, the system can prioritize views that are learning effectively and penalize those that are struggling or overfitting, leading to a more robust and cohesive multi-view representation.
  • Self-Ensembling for Extra Boost: The self-ensemble approach, which leverages the weighted outputs of all classifiers (single-view and multi-view), provides an additional performance improvement, demonstrating the value of each individual component.

Key Results

The proposed multi-view learning method consistently outperformed both single-view baselines and traditional multi-view baselines (concatenation and late fusion) across all three tested tasks.

  • Accuracy Gains: The multi-view network achieved accuracy gains of 0.85% on ESC-50, 1.28% on DCASE2016, and 1.1% on GTZAN over the best respective baselines.
  • Self-Ensemble Improvement: The self-ensemble technique further boosted performance, achieving absolute gains of 1.55% (ESC-50), 1.53% (DCASE2016), and 1.4% (GTZAN).
  • Qualitative Observation: Figure 2(a) and (b) visually demonstrate how the weights adapt during training. For instance, initially, 2D branches (Mel, Gammatone) were given higher weights as they converged faster, but as they started degenerating, their weights were reduced, and the weight for the 1D branch (Raw) was increased as it continued to learn effectively.

Limitations and Future Work

  • Fixed Architecture: The authors acknowledge that their primary goal was to demonstrate the efficacy of their multi-view learning method rather than optimizing the network architecture for each specific task. They did not explore task-specific architectural tailoring or multi-channel combinations, suggesting these could be avenues for further improvement.
  • Comparison Scope: While their method outperforms common baselines, they note that better results have been reported for these datasets by other works, often using different architectures or more specialized techniques. Integrating their proposed multi-view method into state-of-the-art networks is suggested as a promising direction for future research.
  • Computational Cost: While not explicitly stated as a limitation, the multi-view approach with multiple subnetworks and individual classification branches inherently increases computational complexity compared to single-view models.

Summary updated

Abstract

We propose in this work a multi-view learning approach for audio and music classification. Considering four typical low-level representations (i.e. different views) commonly used for audio and music recognition tasks, the proposed multi-view network consists of four subnetworks, each handling one input types. The learned embedding in the subnetworks are then concatenated to form the multi-view embedding for classification similar to a simple concatenation network. However, apart from the joint classification branch, the network also maintains four classification branches on the single-view embedding of the subnetworks. A novel method is then proposed to keep track of the learning behavior on the classification branches and adapt their weights to proportionally blend their gradients for network training. The weights are adapted in such a way that learning on a branch that is generalizing well will be encouraged whereas learning on a branch that is overfitting will be slowed down. Experiments on three different audio and music classification tasks show that the proposed multi-view network not only outperforms the single-view baselines but also is superior to the multi-view baselines based on concatenation and late fusion.
BibTeX
@inproceedings{phan2021multi,
  author = {Huy Phan and Huy Le Nguyen and Oliver Y. Chén and Lam Dang Pham and Philipp Koch and Ian McLoughlin and Alfred Mertins},
  title = {{Multi-view Audio and Music Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=H7uJl7Wm0G},
}
Figure