A Multi-Head Relevance Weighting Framework for Learning Raw Waveform Audio Representations

Debottam Dutta, Purvi Agrawal, Sriram Ganapathy

Abstract

Traditional audio classification systems rely heavily on hand-engineered features like mel-spectrograms, which may not be optimal for all acoustic tasks. This paper presents a multi-head relevance weighting framework designed to learn robust representations directly from raw audio waveforms. The framework utilizes a parametric filterbank followed by a multi-head attention-like mechanism that dynamically weights frequency sub-bands based on their relevance to the classification task. Experimental results on the DCASE2020 and UrbanSound8K datasets demonstrate that this approach significantly outperforms both fixed mel-spectrograms and prior learnable filterbank methods, achieving relative accuracy gains of up to 23%.

Introduction

In the field of audio processing, the mel-spectrogram has long served as the standard representation, mimicking the human auditory system's logarithmic frequency perception. However, fixed representations can limit a model's ability to adapt to specific acoustic environments or noise conditions. Recent shifts toward representation learning from raw waveforms aim to overcome these limitations. While models like SincNet have introduced learnable filters, they often lacked the capacity to selectively focus on the most informative components of the resulting time-frequency maps. This work addresses this gap by introducing a learnable weighting mechanism that identifies and emphasizes task-critical acoustic events.

Proposed Framework

The proposed architecture consists of three integrated stages designed to transform raw audio into a highly discriminative representation.

  1. Parametric Filterbank Layer: The front-end utilizes 1-D convolutional kernels parameterized as cosine-modulated Gaussian filters. These filters offer superior time-frequency localization compared to standard Sinc filters, providing a more precise initial decomposition of the audio signal.
  2. Multi-Head Relevance Weighting: The core of the framework is a multi-head mechanism that partitions the learned spectrum into distinct frequency bands (heads). Each head employs a lightweight neural network to generate a temporal weight mask. This mask acts as a "soft attention" filter, scaling features to highlight relevant sounds and suppress background noise.
  3. Skip-Connections and Classification: To ensure stability and preserve original signal information, the weighted features are combined with the initial filterbank output via skip-connections. The final representation is then processed by a high-level classifier, such as a Fully Convolutional Network (FCN) or a CNN-LSTM (CLSTM).
Overview of the proposed multi-head relevance weighting framework.

Experimental Results

The framework was evaluated on two challenging benchmarks: the DCASE2020 Task 1A (acoustic scene classification) and the UrbanSound8K (USC) dataset.

Performance on DCASE2020

On the DCASE2020 dataset, the proposed "Cos-Gauss + 2head-rel." model achieved an accuracy of 72.6%, representing a 10% relative improvement over the mel-spectrogram baseline. As shown in the results table, the model consistently outperformed other learnable representations across various scene categories.

Accuracy comparison of different representations on the DCASE2020 Task 1A dataset.

Performance on UrbanSound8K

The improvements were even more pronounced in the urban sound classification task. The framework reached an average accuracy of 63%, a 23% relative increase over the 52% achieved by mel-spectrograms. The model showed particular strength in identifying complex mechanical sounds like "Drilling" and "Jackhammer," where the relevance weighting could effectively isolate distinctive rhythmic and spectral patterns.

Discussion and Key Insights

  • Optimal Head Configuration: Ablation studies revealed that a 2-head architecture (splitting the spectrum into two halves) provides the best balance between flexibility and learnability. Interestingly, increasing the number of heads beyond a certain point (e.g., to 80 heads) led to performance degradation, suggesting that relevance is best captured over broader sub-bands.
  • High-Frequency Adaptation: Unlike mel-filterbanks that prioritize lower frequencies, the learned filters in this framework showed a higher density in high-frequency regions. This adaptation is crucial for urban environments, where high-pitched sounds often serve as key identifiers.
  • Computational Efficiency: The relevance sub-networks are designed to be extremely lightweight. In the DCASE experiments, the addition of the multi-head front-end increased the total parameter count by only 0.02%, making it a highly efficient enhancement for existing deep learning models.

Conclusion

The multi-head relevance weighting framework demonstrates the power of combining learnable parametric filters with dynamic attention mechanisms for audio processing. By allowing the model to "learn what to listen to," it overcomes the rigidity of traditional features and the noise sensitivity of raw waveform learning. Future research will focus on addressing class imbalances in datasets and exploring the framework's generalizability to speech and bio-acoustic domains.

Summary updated

BibTeX
@inproceedings{dutta2021multi,
  author = {Debottam Dutta and Purvi Agrawal and Sriram Ganapathy},
  title = {{A Multi-Head Relevance Weighting Framework for Learning Raw Waveform Audio Representations}},
  booktitle = {WASPAA},
  year = {2021},
  doi = {10.1109/waspaa52581.2021.9632708},
}
Figure