Abstract
Acoustic scenes are rich and redundant in their content. In this work, we present a spatio-temporal attention pooling layer coupled with a convolutional recurrent neural network to learn from patterns that are discriminative while suppressing those that are irrelevant for acoustic scene classification. The convolutional layers in this network learn invariant features from time-frequency input. The bidirectional recurrent layers are then able to encode the temporal dynamics of the resulting convolutional features. Afterwards, a two-dimensional attention mask is formed via the outer product of the spatial and temporal attention vectors learned from two designated attention layers to weigh and pool the recurrent output into a final feature vector for classification. The network is trained with between-class examples generated from between-class data augmentation. Experiments demonstrate that the proposed method not only outperforms a strong convolutional neural network baseline but also sets new state-of-the-art performance on the LITIS Rouen dataset.
Introduction
Audio Scene Classification (ASC) is the task of teaching a machine to recognize an environment—like a busy airport, a quiet park, or a bustling restaurant—simply by listening. While humans do this effortlessly, it is a significant challenge for AI because audio recordings are often packed with redundant information and irrelevant background sounds that can easily confuse a standard neural network.
Most existing ASC systems struggle with the "richness" of acoustic scenes. A 30-second clip of a train station contains a few key sounds (a whistle, an announcement) buried in a sea of constant, irrelevant rumble. Standard deep learning models, like Convolutional Neural Networks (CNNs), are great at picking up local patterns, but they often struggle to weigh which patterns actually matter over time. While some researchers have used "temporal attention" to find when an important sound happens, they often ignore the "spatial" aspect—which specific frequency features are most indicative of the scene. This paper introduces a spatio-temporal attention pooling mechanism to bridge this gap by looking at both dimensions simultaneously.
Methods
The proposed system, called Att-CRNN, consists of three main stages:
- Multi-Channel Input: The system takes a 2-channel input. The first is a standard log Mel-spectrogram. The second is an "auxiliary channel" created by subtracting background noise, which helps expose foreground events more clearly.
- The CRNN Backbone: Three convolutional layers extract invariant features from the time-frequency input, followed by bidirectional Gated Recurrent Units (GRUs) that model how these features evolve over time.
- Spatio-Temporal Attention Pooling: Instead of simple pooling, the model learns two separate attention vectors: one for time (temporal) and one for features (spatial). By taking the outer product of these two vectors, the model creates a 2D "attention mask." This mask is multiplied by the recurrent output to weight the most important entries before summing them into a final feature vector.
- Between-Class (BC) Augmentation: To prevent overfitting, the authors mixed samples from different classes and trained the network to predict the ratio of the mix using Kullback-Leibler (KL) divergence loss, forcing the network to learn more robust decision boundaries.
Experimental Results
The model was evaluated on the LITIS Rouen dataset, a benchmark for ASC containing 3,026 examples across 19 scene categories. The Att-CRNN significantly outperformed both its own CNN baseline and previous state-of-the-art (SOTA) models.
Key insights from the results include:
- SOTA Performance: The system achieved a top accuracy of 98.72% on the LITIS Rouen dataset, surpassing the previous best of 98.06%.
- Error Reduction: Compared to a standard CNN baseline, the proposed attention mechanism reduced the classification error by roughly 22% to 27% depending on the features used.
- Robustness: By adding a spatial dimension to the attention mechanism, the model can ignore specific frequency bands that are noisy across the entire clip, leading to more stable performance during training.
Conclusion
The integration of spatio-temporal attention pooling allows neural networks to more effectively filter the redundant information inherent in acoustic scenes. By focusing on discriminative patterns in both time and frequency, the Att-CRNN sets a new performance benchmark. Future work could involve testing this pooling mechanism on more diverse datasets and exploring its potential for model explainability through mask visualization.
Summary updated