Divided spectro-temporal attention for sound event localization and detection in real scenes for DCASE2023 challenge

Yusun Shul, Byeong-Yun Ko, Jung‐Woo ChoiarXiv 2023 · DOI · arXiv

TL;DR

Researchers from KAIST have developed a "Divided Spectro-Temporal (DST) Attention" mechanism to improve how AI localizes and identifies sounds in real-world environments. By processing spectral (frequency) and temporal (time) information through separate attention layers and utilizing aggressive data augmentation, their model significantly reduces overfitting to specific room acoustics, outperforming the DCASE2023 challenge baseline.

What Problem Does This Paper Address?

Sound Event Localization and Detection (SELD) is the "double-whammy" of audio processing: the AI must identify what a sound is (detection) and where it is coming from (localization).

The biggest hurdle in real-world SELD is reverberation. Every room has a unique "acoustic fingerprint" based on its shape and materials. When deep learning models are trained on recordings from just a few rooms, they tend to overfit—learning the specific echoes of those rooms rather than the general characteristics of the sounds themselves. Standard architectures (like CRNNs) often collapse frequency information too early or focus primarily on temporal patterns, missing the nuanced spectral details that could help the model generalize across different environments.

Methods

The authors introduce a novel architecture called Divided Spectro-Temporal (DST) Attention.

  1. Feature Extraction: The model takes 7-channel input (4 from First-Order Ambisonics spectrograms and 3 from intensity vectors) to capture both the "what" and the "where."
  2. Reduced Frequency Pooling: Unlike the baseline model, which aggressively shrinks the frequency dimension, this model keeps more spectral detail (reducing it to a dimension of 16 instead of 2).
  3. Divided Attention: Instead of one giant attention layer, they split the work:
  4. S-MHSA (Spectral Multi-Head Self-Attention): This layer focuses purely on the relationships between different frequency bands.
  5. T-MHSA (Temporal Multi-Head Self-Attention): This layer focuses on how the sound evolves over time.
  6. Data Augmentation: To further combat overfitting, they used a "kitchen sink" approach to augmentation:
  7. Frameshift: Shifting features and labels along the time axis.
  8. Time Masking: Randomly "blacking out" chunks of time.
  9. Channel Swapping: A spatial augmentation that swaps audio channels to simulate different microphone orientations.
  10. Moderate Mix-up: Blending two different soundscapes together to create more complex training examples.

Datasets

  • STARSS23: The primary dataset containing real-world recordings with spatial annotations.
  • AudioSet: Used as external data. The authors synthesized 6 hours of additional training data by convolving AudioSet samples with TAU-SRIR (Spatial Room Impulse Responses) to create realistic spatial audio.
  • FSD50k: Used for additional synthetic data mixtures.

Key Insights and Ideas

  • The "Divided" Advantage: Borrowing a concept from video classification (where space and time are often processed separately), the authors found that treating frequency and time as independent axes for attention allows the model to learn more robust features. It prevents the temporal context from "washing out" important spectral markers.
  • The Resolution Sweet Spot: There is a trade-off in frequency resolution. The authors experimented with different spectral dimensions ($F'$). While the baseline uses $F'=2$, they found that $F'=16$ provided the best performance. Interestingly, increasing resolution further (to 32 or 64) actually caused performance to drop, suggesting that too much spectral detail can become "noise" that leads back to overfitting.
  • Augmentation Synergy: The combination of spatial augmentation (channel swapping) and external data was crucial. It forced the model to rely on the physics of sound rather than the specific quirks of the training recordings.

Key Results

The DST Attention model showed clear improvements over the DCASE2023 baseline across all primary metrics:

  • SELD Score: Improved from 0.479 (baseline) to 0.407 (best DST model with augmentation). Note: for SELD scores, lower is better.
  • Error Rate (ER): Reduced from 0.570 to 0.495.
  • F-score (Detection): Increased from 29.9% to 42.7%.
  • Localization Error (LE): Decreased from 22.0° to 16.5°.

The results demonstrate that the "Single-ACCDOA" format, when combined with DST attention and external data, yielded the most accurate results for real-world testing.

Limitations and Future Work

The authors noted a specific limitation regarding the spectral dimension size: while increasing resolution to a certain point ($F'=16$) helped, going beyond that yielded negative returns for single-ACCDOA training. This suggests the model may struggle to attend to very high-dimensional spectral maps effectively.

Future work could involve exploring more sophisticated ways to fuse the spectral and temporal branches or investigating if different attention head configurations could handle higher spectral resolutions without performance degradation.

Summary updated

Abstract

Localizing sounds and detecting events in different room environments is a difficult task, mainly due to the wide range of reflections and reverberations. When training neural network models with sounds recorded in only a few room environments, there is a tendency for the models to become overly specialized to those specific environments, resulting in overfitting. To address this overfitting issue, we propose divided spectro-temporal attention. In comparison to the baseline method, which utilizes a convolutional recurrent neural network (CRNN) followed by a temporal multi-head self-attention layer (MHSA), we introduce a separate spectral attention layer that aggregates spectral features prior to the temporal MHSA. To achieve efficient spectral attention, we reduce the frequency pooling size in the convolutional encoder of the baseline to obtain a 3D tensor that incorporates information about frequency, time, and channel. As a result, we can implement spectral attention with channel embeddings, which is not possible in the baseline method dealing with only temporal context in the RNN and MHSA layers. We demonstrate that the proposed divided spectro-temporal attention significantly improves the performance of sound event detection and localization scores for real test data from the STARSS23 development dataset. Additionally, we show that various data augmentations, such as frameshift, time masking, channel swapping, and moderate mix-up, along with the use of external data, contribute to the overall improvement in SELD performance.
BibTeX
@misc{shul2023divided,
  author = {Yusun Shul and Byeong-Yun Ko and Jung‐Woo Choi},
  title = {{Divided spectro-temporal attention for sound event localization and detection in real scenes for DCASE2023 challenge}},
  year = {2023},
  eprint = {2306.02591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.02591},
}
Figure