SRP-DNN: Learning Direct-Path Phase Difference for Multiple Moving Sound Source Localization

Bing Yang, Hong Liu, Xiaofei Li

TL;DR

This paper introduces SRP-DNN, a novel deep learning approach for localizing multiple moving sound sources in challenging real-world conditions. By learning direct-path phase differences, SRP-DNN significantly improves localization accuracy and robustness against noise and reverberation, outperforming existing methods.

What Problem Does This Paper Address?

Localizing multiple sound sources that are moving simultaneously is a notoriously difficult problem in acoustics. Real-world scenarios are plagued by issues like sound sources interacting with each other, their trajectories constantly changing, and the presence of background noise and echoes (reverberation). Traditional methods, while effective for simpler cases, struggle to accurately pinpoint multiple moving sources in these complex environments. Existing deep learning approaches often face challenges with ambiguous source assignment and variable output dimensions when dealing with multiple targets. This paper aims to bridge this gap by developing a robust method that can effectively disentangle and track multiple moving sound sources in real-time, even under adverse acoustic conditions.

Methods

The core innovation of SRP-DNN lies in its use of a Causal Convolutional Recurrent Neural Network (CRNN) to learn direct-path phase differences (IPDs). Here's a breakdown:

  • Direct-Path Phase Differences (IPDs): The authors leverage the fact that the phase difference between signals received by two microphones is directly related to the time difference of arrival (TDOA) of a sound source. They focus on the "direct-path" component, which is the sound traveling directly from the source to the microphone, as it's less affected by reflections.
  • Causal CRNN Architecture: This network is designed to process sequential audio data.
  • Convolutional layers: These layers extract local temporal and frequency patterns from the audio signals, capturing short-term context.
  • Recurrent layers (GRU): These layers process the output of the convolutional layers to capture longer-term temporal dependencies, crucial for tracking moving sources.
  • Causality: The network is "causal," meaning it only uses past and present information to make predictions, which is essential for real-time applications.
  • Novel Learning Target: A key challenge in multi-source localization is assigning the correct IPD to each source. Instead of trying to predict individual IPDs (which leads to assignment ambiguity), SRP-DNN learns a weighted sum of direct-path IPDs. The "weight" in this sum represents the activity of each source. This clever formulation allows the network to learn a combined representation that can be directly used to construct a spatial spectrum.
  • Iterative Source Detection and Localization (IDL): After the CRNN predicts the weighted IPDs, these are used to form an enhanced spatial spectrum (SRP-DNN). To handle overlapping sources, the paper proposes an iterative approach:
  • Detect the strongest peak in the spatial spectrum (representing the dominant source).
  • Estimate its DOA.
  • "Remove" the contribution of this dominant source from the spatial spectrum.
  • Repeat until no significant peaks remain. This iterative removal helps to disentangle sources that might otherwise merge.

Datasets

  • Simulated Data: Generated using room impulse responses (RIRs) created with the gpuRIR library, simulating various room sizes and reverberation times. Speech recordings were sourced from the LibriSpeech corpus.
  • LOCATA Dataset: Used for evaluation, including both simulated and real-world recordings. The real-world data comes from a computing laboratory and includes scenarios with single and multiple moving sources.

Key Insights and Ideas

  • Learning IPDs is More Robust: Focusing on learning direct-path IPDs, rather than raw acoustic features, proves to be a more robust strategy for localization, especially in noisy and reverberant conditions.
  • Weighted Sum for Ambiguity Resolution: The proposed learning target, a weighted sum of IPDs, elegantly sidesteps the assignment ambiguity problem inherent in multi-target regression tasks.
  • CRNN for Spatio-Temporal Context: The CRNN architecture effectively captures the temporal smoothness and frequency-linear properties of direct-path IPDs, crucial for tracking moving sources.
  • Iterative Removal for Source Separation: The IDL strategy is a simple yet effective way to disentangle overlapping sources in the spatial spectrum, improving the accuracy of individual source localization.

Key Results

  • Improved Robustness: SRP-DNN demonstrates superior performance in noisy and reverberant environments compared to traditional SRP-PHAT. Figure 2 visually shows how SRP-DNN's spatial spectra maintain clear peaks while SRP-PHAT's become indistinct.
  • State-of-the-Art Performance on LOCATA: Table 1 shows that SRP-DNN, particularly with the IDL method, achieves significantly lower Miss Detection Rates (MDR) and False Alarm Rates (FAR) for multi-source localization compared to baseline methods like Cross3D, CTF-DPRTF, and SRP-PHAT.
  • Accurate Trajectory Estimation: Figure 3 provides a qualitative example showing that SRP-DNN+IDL produces fewer erroneous DOA estimates (outliers) and more correct estimates for two moving sources.

Limitations and Future Work

  • Kmax Limitation: The network is designed to handle up to a predefined maximum number of sources (Kmax). While Kmax was set to 2 for experiments, extending this to a truly unknown number of sources would be a valuable future direction.
  • Thresholding in IDL: The iterative source detection relies on a threshold (βTH) to determine when a source is no longer significant. The selection of this threshold can impact performance, and further research into adaptive thresholding could be beneficial.
  • Real-time Performance: While the causal nature of the CRNN facilitates online processing, the computational complexity of the CRNN and the iterative localization process might need further optimization for very high-frequency real-time applications.

Summary updated

Abstract

Multiple moving sound source localization in real-world scenarios remains a challenging issue due to interaction between sources, time-varying trajectories, distorted spatial cues, etc. In this work, we propose to use deep learning techniques to learn competing and time-varying direct-path phase differences for localizing multiple moving sound sources. A causal convolutional recurrent neural network is designed to extract the direct-path phase difference sequence from signals of each microphone pair. To avoid the assignment ambiguity and the problem of uncertain output-dimension encountered when simultaneously predicting multiple targets, the learning target is designed in a weighted sum format, which encodes source activity in the weight and direct-path phase differences in the summed value. The learned direct-path phase differences for all microphone pairs can be directly used to construct the spatial spectrum according to the formulation of steered response power (SRP). This deep neural network (DNN) based SRP method is referred to as SRP-DNN. The locations of sources are estimated by iteratively detecting and removing the dominant source from the spatial spectrum, in which way the interaction between sources is reduced. Experimental results on both simulated and real-world data show the superiority of the proposed method in the presence of noise and reverberation.
BibTeX
@inproceedings{yang2022srp,
  author = {Bing Yang and Hong Liu and Xiaofei Li},
  title = {{SRP-DNN: Learning Direct-Path Phase Difference for Multiple Moving Sound Source Localization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746624},
}
Figure