Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting

Purvi Agrawal, Sriram Ganapathy

Abstract

This paper introduces a novel, biologically-inspired architecture for speech and audio signal processing that utilizes a two-stage relevance weighting mechanism. By learning representations directly from raw waveforms through acoustic and modulation filterbank layers, the model achieves superior performance in noisy and reverberant environments compared to traditional Mel-spectrogram baselines. Crucially, the architecture offers enhanced interpretability by allowing for the analysis of learned relevance weights, which correspond to phonetic properties and environmental adaptations.

Introduction

For decades, the Mel-spectrogram has served as the "gold standard" input for speech recognition. However, these fixed, hand-engineered features are often suboptimal for varied environments, such as noisy construction sites or reverberant halls. While recent trends in "Representation Learning" attempt to learn features directly from raw audio, these models are frequently "black boxes," making it difficult to understand what the network is actually processing. This paper addresses the dual challenge of performance and interpretability by proposing a two-stage architecture inspired by the human auditory system.

Methods

The proposed architecture mimics the two-stage processing of the human ear and brain:

  1. Acoustic Filterbank Layer (The "Cochlea"): The first layer utilizes parametric 1D filters (cosine-modulated Gaussian functions) instead of standard convolutions. A "relevance sub-network" performs soft feature selection by assigning weights to each filter, effectively highlighting important frequency ranges for specific phonemes.
  2. Modulation Filterbank Layer (The "Auditory Cortex"): The output from the first stage enters a 2D convolutional layer to capture spectro-temporal patterns. A second relevance sub-network then weights these modulation features.
  3. Relevance Weighting: Unlike standard attention mechanisms, this weighting acts as a gain control or gate, modulating information flow based on the input's content.
Architecture of the proposed CNN-DNN with relevance weighting sub-networks.

The entire system is differentiable and trained end-to-end using standard backpropagation.

Results and Analysis

The model was rigorously tested across several benchmarks, including Aurora-4 (noisy speech), CHiME-3 (reverberant speech), and UrbanSound8K (environmental sounds).

Performance and Efficiency

On the Aurora-4 dataset, the proposed model achieved a Word Error Rate (WER) of 9.6%, a significant improvement over the 10.7% achieved by standard Mel-filterbank (MFB) baselines. In the UrbanSound8K task, the model reached 62% accuracy, outperforming the MFB baseline of 52%. Despite the added interpretability, the model only increases the total parameter count by approximately 0.2%.

Interpretability and Bio-Mimicry

Analysis of the relevance weights revealed that the model "learned" phonetic properties. For example, for "front" vowels like /iy/, the model automatically assigned higher weights to frequency regions corresponding to the first and second formants (F1 and F2). Furthermore, the model exhibits "adaptive listening"; in high-noise conditions, relevance weights become more uniform, mirroring the degradation of human auditory segregation in loud environments.

Comparison of learned center frequencies for Aurora-4 and CHiME-3 datasets against the standard Mel-scale.

As shown in the figure above, the learned filterbank allocates more filters to lower frequencies than a standard Mel-scale, which was found to be key for handling reverberation in datasets like CHiME-3.

Conclusion

This work demonstrates that bio-mimetic structures can lead to more robust and interpretable speech representation models. By moving away from "black box" deep learning, researchers can gain insights into the acoustic features deemed relevant by the model. Future work will explore the implementation of "top-down" feedback mechanisms and the application of these features to tasks such as speaker verification and emotion recognition.

Summary updated

BibTeX
@article{agrawal2020interpretable,
  author = {Purvi Agrawal and Sriram Ganapathy},
  title = {{Interpretable Representation Learning for Speech and Audio Signals Based on Relevance Weighting}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2020},
  doi = {10.1109/taslp.2020.3030489},
}
Figure