SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays

Thi Ngoc Tho Nguyen, Douglas L. Jones, Karn N. Watcharasupat, Huy Phan, Woon-Seng Gan

TL;DR

This paper introduces SALSA-Lite, a new, computationally efficient feature for real-time sound event localization and detection (SELD) using microphone arrays. By replacing complex eigenvector calculations with simpler inter-channel phase differences, SALSA-Lite achieves a 30x speedup over its predecessor while maintaining competitive performance, significantly outperforming existing standard features.

What Problem Does This Paper Address?

Sound Event Localization and Detection (SELD) is crucial for understanding acoustic environments, enabling applications like smart assistants and surveillance. The challenge lies in simultaneously identifying what sounds are present (detection), when they start and stop (event timing), and where they are coming from (localization). Polyphonic SELD, dealing with multiple overlapping sounds, is particularly difficult. A major bottleneck for real-time SELD systems is the high computational cost of extracting features that capture both spectral and spatial information from microphone array data. Existing state-of-the-art features, while effective, are too slow for real-time applications, necessitating the development of faster, yet still accurate, feature extraction methods.

Methods

The paper proposes SALSA-Lite, a novel feature designed to be computationally efficient for SELD with microphone arrays. It's a variation of the SALSA (Spatial Cue-Augmented Log-Spectrogram) feature.

  • SALSA: This original feature combines multichannel log-spectrograms with Eigenvector-based Phase Vectors (EPVs). EPVs are derived from the principal eigenvectors of spatial covariance matrices, capturing spatial information. However, computing these eigenvectors is computationally intensive.
  • SALSA-Lite: This new feature replaces the EPVs with Normalized Inter-Channel Phase Differences (NIPDs). NIPDs are calculated directly from the complex spectrograms and are a simpler, faster approximation of spatial cues. The authors argue that for a single dominant sound source in a time-frequency bin, NIPDs can approximate the relative distances of arrival (RDOAs), similar to EPVs. This simplification leads to a significant speedup.
  • Data Augmentation: To improve robustness and performance, the authors employ three data augmentation techniques: channel swapping (CS), random cutout (RC), and frequency shifting (FS).

Datasets

The experiments were conducted on the TAU-NIGENS Spatial Sound Events (TNSSE) 2021 dataset. This dataset provides multichannel audio clips (specifically, the MIC format with 4 channels) for training, validation, and testing, featuring 12 distinct sound classes. The dataset is characterized by reverberant conditions and the presence of directional interferers.

Key Insights and Ideas

  • The Trade-off Between Complexity and Performance: The core insight is that simpler spatial features (NIPDs) can achieve performance comparable to more complex ones (EPVs) for SELD, especially when combined with effective deep learning models. This highlights that the "best" feature isn't always the most mathematically complex.
  • The Importance of TF Alignment: SALSA-Lite, like its predecessor SALSA, maintains an exact time-frequency (TF) alignment between spectral and spatial information. This is crucial for CNN-based models, which can more effectively learn patterns from image-like feature representations by processing local TF patches. This contrasts with features like GCC-PHAT, where TF alignment is less direct.
  • Data Augmentation's Significant Impact: The study demonstrates that data augmentation techniques provide substantial performance gains across all tested features, particularly benefiting the traditional MELSPECGCC. This underscores the importance of robust training strategies for SELD.
  • NIPD's Effectiveness: The results show that the normalized inter-channel phase differences (NIPD) in SALSA-Lite are effective for capturing spatial cues, even outperforming the unnormalized SALSA-IPD feature. This suggests that the frequency normalization plays a beneficial role.

Key Results

  • Speedup: SALSA-Lite offers a remarkable 30-fold speedup in feature computation compared to the original SALSA feature. It is also 9 times faster than the traditional MELSPECGCC feature.
  • Performance:
    • SALSA-Lite achieved competitive performance with the full SALSA feature.
  • It significantly outperformed the traditional MELSPECGCC feature. Specifically, SALSA-Lite increased the localization-dependent F1 score by 15% and class-dependent localization recall by 5% compared to MELSPECGCC.
  • With data augmentation, SALSA-Lite achieved an error rate (ER<20°) of 0.409 and an F1 score (F<20°) of 0.707 on the test set.
  • Robustness: SALSA-Lite and SALSA-IPD were found to be only mildly affected by spatial aliasing, performing similarly with cutoff frequencies of 2kHz and 9kHz.

Limitations and Future Work

  • Spatial Aliasing: While SALSA-Lite is less affected by spatial aliasing than some other methods, the paper notes that performance can be slightly better with a lower cutoff frequency (2kHz) for NIPD and IPD.
  • Ensemble Models: The paper compares its single CRNN model trained on SALSA-Lite against state-of-the-art systems that often use ensemble models. While SALSA-Lite performs competitively, further improvements might be possible by integrating it into ensemble frameworks.
  • Future Work: The authors suggest that SALSA-Lite is a promising candidate for real-time SELD. They also highlight that various data augmentation techniques can be applied on-the-fly during training to further enhance model performance, implying that exploring more advanced augmentation strategies could be beneficial.

Summary updated

Abstract

Polyphonic sound event localization and detection (SELD) has many practical applications in acoustic sensing and monitoring. However, the development of real-time SELD has been limited by the demanding computational requirement of most recent SELD systems. In this work, we introduce SALSA-Lite, a fast and effective feature for polyphonic SELD using microphone array inputs. SALSA-Lite is a lightweight variation of a previously proposed SALSA feature for polyphonic SELD. SALSA, which stands for Spatial Cue-Augmented Log-Spectrogram, consists of multichannel log-spectrograms stacked channel-wise with the normalized principal eigenvectors of the spectrotemporally corresponding spatial covariance matrices. In contrast to SALSA, which uses eigenvector-based spatial features, SALSA-Lite uses normalized inter-channel phase differences as spatial features, allowing a 30-fold speedup compared to the original SALSA feature. Experimental results on the TAU-NIGENS Spatial Sound Events 2021 dataset showed that the SALSA-Lite feature achieved competitive performance compared to the full SALSA feature, and significantly outperformed the traditional feature set of multichannel log-mel spectrograms with generalized cross-correlation spectra. Specifically, using SALSA-Lite features increased localization-dependent F1 score and class-dependent localization recall by 15% and 5%, respectively, compared to using multichannel log-mel spectrograms with generalized cross-correlation spectra.
BibTeX
@inproceedings{nguyen2022salsa,
  author = {Thi Ngoc Tho Nguyen and Douglas L. Jones and Karn N. Watcharasupat and Huy Phan and Woon-Seng Gan},
  title = {{SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=8gnh3oqaEg},
}
Figure