Temporal Attentive Pooling for Acoustic Event Detection

Xugang Lu, Peng Shen, Sheng Li, Yu Tsao, Hisashi Kawai2018 · DOI

TL;DR

This paper introduces a novel "temporal attentive pooling" mechanism for acoustic event detection (AED). By using both global and local attention modules within a Convolutional Recurrent Neural Network (CRNN), the model learns to dynamically weigh the importance of different temporal frames, leading to improved accuracy in identifying and classifying acoustic events, especially when event annotations are not precisely time-stamped.

What Problem Does This Paper Address?

Acoustic Event Detection (AED) aims to identify and classify specific sounds within audio streams, like a car horn, a dog bark, or a siren. Traditional deep learning approaches for AED often rely on pooling strategies (like average or max pooling) to condense temporal features into a single representation for classification. However, these methods assume that either the entire segment or a single peak frame is representative of the event. This is problematic because acoustic events are often described holistically, making it difficult to pinpoint exact temporal locations that contribute most to the perception of an event. This ambiguity, especially in weakly supervised scenarios where only clip-level labels are available, can lead to confusion and reduced classification performance. The paper addresses this by proposing a more nuanced way to aggregate temporal features, acknowledging that not all frames are equally important for event detection.

Methods

The core of the proposed method is a Convolutional Recurrent Neural Network (CRNN) enhanced with global and local attention mechanisms.

  • CRNN Backbone: A standard CRNN architecture is used, where Convolutional Neural Networks (CNNs) extract frequency-shift invariant features, and Recurrent Neural Networks (RNNs, specifically LSTMs) model the temporal dependencies within these features.
  • Temporal Attentive Pooling: Instead of fixed pooling, the paper introduces attention modules to dynamically weight temporal features.
  • Global Attention (GAT): This module takes both the CNN output at a given time step and the global representation from the RNN as input. It learns an attention weight that reflects the importance of that time step in the context of the entire sequence.
  • Local Attention (LAT): This module focuses solely on the CNN output at a given time step and learns a local attention weight, refining the importance of that specific frame.
  • Feature Aggregation: The final feature representation is derived by combining the attention-weighted CNN features (using both global and local attention weights) and the global feature extracted by the RNN. This allows the model to focus on discriminative frames while still leveraging the overall temporal context.

Datasets

The experiments were conducted using the DCASE (Detection and Classification of Acoustic Scenes and Events) 2016 challenge dataset. This dataset includes audio recordings from "Home" and "Residential area" environments, with 18 audio event categories. The authors used four experimental folds for evaluation, each with distinct training, validation, and testing sets.

Key Insights and Ideas

  • Beyond Fixed Pooling: The paper highlights the limitations of traditional average and max pooling for AED, especially when dealing with holistic event descriptions and label uncertainty.
  • Attention as a Soft Measure: Attention mechanisms are used not just for feature selection but as a probabilistic way to estimate the "event presence likelihood" (EPL) for each frame. This allows for a more nuanced aggregation of temporal information.
  • Synergy of Global and Local Attention: Combining both global and local attention modules proved more effective than using either one in isolation. Global attention captures broader temporal context, while local attention refines frame-level importance.
  • Leveraging Label Uncertainty: The proposed attention framework is particularly well-suited for scenarios with weakly supervised learning or label uncertainty, where precise temporal annotations are scarce.

Key Results

The proposed "GL_ATT_CAT" model, which concatenates attention-weighted features with the RNN's global representation, achieved the best performance among the tested models. Specifically, it reached a segment-based F1 score of 45.2% on average across all folds, outperforming baseline models like GMM, RNN, and CNN. The paper also notes that integrating either global or local attention individually improved performance, but their combination yielded the most significant gains.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the attention mechanism helps, the fundamental challenge of label uncertainty in weakly supervised learning remains. The performance is still dependent on the quality of the provided clip-level labels.
  • Future Work: The authors suggest that this attention-controlled temporal pooling framework is highly applicable to other tasks involving label uncertainty or where precise annotations are difficult to obtain. They plan to extend this approach to other related audio and potentially other signal processing tasks.

Summary updated

BibTeX
@misc{lu2018temporal,
  author = {Xugang Lu and Peng Shen and Sheng Li and Yu Tsao and Hisashi Kawai},
  title = {{Temporal Attentive Pooling for Acoustic Event Detection}},
  year = {2018},
  doi = {10.21437/interspeech.2018-1552},
}
Figure