Spatio-Temporal Attention Pooling for Audio Scene Classification

Huy Phan, Oliver Y. Chén, Lam Dang Pham, Philipp Koch, Maarten De Vos, Ian McLoughlin, Alfred Mertins

TL;DR

This paper introduces a Convolutional Recurrent Neural Network (CRNN) that pairs bidirectional recurrent layers with a joint spatio-temporal attention pooling mechanism for Audio Scene Classification (ASC). By taking the outer product of independent temporal and feature-wise (spatial) attention vectors, the model suppresses background noise and selectively highlights discriminative acoustic patterns. Combined with between-class data augmentation, the method establishes a new state-of-the-art accuracy of 98.72% on the LITIS Rouen dataset.

What Problem Does This Paper Address?

Audio scenes (such as busy train stations, markets, or cafes) are cluttered, continuous mixtures of persistent background noise and transient foreground events. Because acoustic recordings contain substantial irrelevant and redundant information, standard deep learning models—such as vanilla Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs)—frequently overfit to background textures.

While temporal soft attention has been used to focus on when key audio events happen, it ignores which learned feature dimensions (the spatial axis) are most informative. This paper tackles the challenge of jointly discovering and amplifying discriminative cues across both time and feature dimensions simultaneously.

Methods

The proposed architecture combines signal pre-processing, sequential deep feature extraction, and a custom pooling mechanism:

2-channel Audio (Raw + Noise-Subtracted) 
   → Log Spectrogram (M × T × 2)
   → 3× Conv + MaxPool (Downsamples frequency, preserves time T)
   → Bidirectional GRU (Yields hidden representations Z ∈ R^{2H × T})
   → Dual Attention Layers:
         • Temporal vector (a^tem ∈ R^T)
         • Spatial/Feature vector (a^spa ∈ R^{2H})
   → Outer Product: Spatio-Temporal Mask A = a^spa ⊗ a^tem
   → Tanh-weighted Temporal Pooling → Linear SVM Classifier

Key technical elements include:

  • Two-Channel Audio Input: Alongside the standard time-frequency spectrogram (log Mel or log Gammatone), the authors create a second auxiliary channel by estimating and subtracting background noise using minimum statistics.
  • CRNN Feature Extractor: Three convolutional layers with 1D max pooling compress frequency dimensions while preserving sequence length $T$. Stood atop these are stacked Bidirectional Gated Recurrent Units (BiGRUs) that model temporal dynamics.
  • Spatio-Temporal Attention Pooling: Rather than pooling across time uniformly or with 1D temporal attention alone, the model computes:
  • A temporal attention vector $a^{\text{tem}} \in \mathbb{R}^T$.
  • A spatial attention vector $a^{\text{spa}} \in \mathbb{R}^{2H}$ across the recurrent hidden dimensions.
  • A 2D attention mask via outer product: $A = a^{\text{spa}} \otimes a^{\text{tem}}$.
  • The pooled vector $x$ is produced by summing over time with a hyperbolic tangent activation: $x_s = \sum_{t=1}^T \tanh(A_{st} Z_{st})$, which both bounds weights and amplifies salient signals.
  • Between-Class (BC) Augmentation & KL-Divergence Loss: Training pairs from different classes are mixed linearly with a random ratio $r \in (0, 1)$, and the model is trained against the mixed soft-target distribution using Kullback-Leibler (KL) divergence.
  • SVM Calibration: At test time, embeddings extracted by the network are classified using a linear Support Vector Machine (SVM) calibrated to produce posterior probabilities, exploiting SVMs' maximum-margin generalization.

Datasets

  • LITIS Rouen: A benchmark dataset consisting of 3,026 30-second recordings across 19 urban acoustic scene classes (totaling 1,500 minutes of audio sampled at 22,050 Hz).
  • Evaluation follows the standard protocol: 20 random train/test splits, segmenting 30-second audio clips into non-overlapping 2-second segments and combining segment-level probabilities via probabilistic multiplicative fusion.
  • No new datasets were introduced.

Key Insights and Ideas

  • Factored 2D Attention Avoids Parameter Explosion: Directly learning a full $2H \times T$ attention matrix can be parameter-heavy and prone to overfitting. Decoupling the scoring into distinct 1D spatial and 1D temporal scoring layers—then taking their outer product—provides entry-level weighting across the entire feature-time grid with minimal parameter overhead.
  • $\tanh$ Prior to Summation: In contrast to standard weighted averaging, applying a $\tanh$ non-linearity to the element-wise masked representations ($A \odot Z$) helps actively suppress near-zero noise while boosting large positive/negative feature responses.
  • Multi-View Spectrogram Complementarity: Log Mel-scale and log Gammatone filterbanks capture acoustic frequencies differently. Fusing predictions from models trained independently on each representation yielded noticeable performance bumps over either alone.

Key Results

  • Segment-Level Performance (2-second clips): Att-CRNN outperformed the deep CNN baseline by +1.45% (93.78% vs. 92.33%) on log Mel-scale features and +1.52% (93.65% vs. 92.13%) on log Gammatone features.
  • Snippet-Level Performance (30-second clips): Att-CRNN achieved 98.53% (log Mel) and 98.46% (log Gammatone), representing relative error reductions of 22.22% and 27.70% over the CNN baseline.
  • State-of-the-Art Comparison:
  • Late fusion of log Mel and log Gammatone Att-CRNN models achieved 98.72% accuracy (98.57% F1-score, 98.40% precision).
  • This surpassed the prior state-of-the-art—a Temporal Transformer CNN (98.06%)—cutting the relative classification error by 34.02%.

Limitations and Future Work

  • Single Benchmark Evaluation: Experiments were restricted exclusively to the LITIS Rouen dataset; validation across broader challenge benchmarks (e.g., DCASE series) is needed to assess robustness across varying acoustic environments and hardware recording devices.
  • Multi-Stage Pipeline Complexity: The pipeline relies on multiple disconnected phases: offline minimum-statistics noise subtraction, BC training with a neural net, feature extraction, linear SVM calibration, and multi-segment late fusion. Consolidating this into an end-to-end framework remains an open engineering step.
  • Computational Cost: Stacking bidirectional GRUs over time frames adds latency and memory overhead relative to pure feed-forward CNNs, which could limit deployment on low-power edge sensors.

Summary updated

Abstract

Acoustic scenes are rich and redundant in their content. In this work, we present a spatio-temporal attention pooling layer coupled with a convolutional recurrent neural network to learn from patterns that are discriminative while suppressing those that are irrelevant for acoustic scene classification. The convolutional layers in this network learn invariant features from time-frequency input. The bidirectional recurrent layers are then able to encode the temporal dynamics of the resulting convolutional features. Afterwards, a two-dimensional attention mask is formed via the outer product of the spatial and temporal attention vectors learned from two designated attention layers to weigh and pool the recurrent output into a final feature vector for classification. The network is trained with between-class examples generated from between-class data augmentation. Experiments demonstrate that the proposed method not only outperforms a strong convolutional neural network baseline but also sets new state-of-the-art performance on the LITIS Rouen dataset.
BibTeX
@inproceedings{phan2019spatio,
  author = {Huy Phan and Oliver Y. Chén and Lam Dang Pham and Philipp Koch and Maarten De Vos and Ian McLoughlin and Alfred Mertins},
  title = {{Spatio-Temporal Attention Pooling for Audio Scene Classification}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=M5wdj0l6PN},
}

Loading related content...

Figure