Robust DOA Estimation Based on Convolutional Neural Network and Time-Frequency Masking

Wangyou Zhang, Ying Zhou, Yanmin QianInterspeech 2019 · OpenReview

Abstract

In scenarios with noise and reverberation, the performance of current methods for direction of arrival (DOA) estimation usually degrades significantly. Inspired by the success of time-frequency masking in speech enhancement and speech separation, this paper proposes new methods to better utilize time-frequency masking in convolutional neural networks to improve the robustness of localization. First, a mask estimation network is developed to assist DOA estimation by either appending or multiplying the estimated masks to the original input feature. Then, a multi-task learning architecture is proposed to optimize the mask and DOA estimation networks jointly. Experiments show that all the proposed methods have better robustness and generalization in noisy and reverberant conditions compared to conventional methods, with multi-task methods achieving the best performance.

Introduction

Determining the Direction of Arrival (DOA) of sound is a foundational task for robotics, smart speakers, and teleconferencing. While classical algorithms like MUSIC or SRP-PHAT perform well in controlled environments, they often fail in real-world settings where background noise and reverberation (echoes) distort the signal. Recent deep learning approaches have improved performance, but they often treat all parts of a sound signal equally, failing to distinguish between target speech and noise. This paper addresses the challenge by teaching a neural network to focus on informative time-frequency (T-F) bins.

Proposed Methods

The authors propose three architectures based on a baseline CNN that utilizes the phase component of the Short-Time Fourier Transform (STFT) as the primary input feature:

  1. Mask-CNN: A separate mask estimation network is trained to predict an Ideal Ratio Mask (IRM). This mask highlights T-F bins where speech is dominant. The mask is then either appended to or multiplied by the DOA network's input to "clean" the features.
  2. Standard Multi-task Learning (MTL): The mask and DOA networks are connected and trained simultaneously. This ensures the mask is optimized specifically for localization.
  3. Pseudo Multi-task Learning: In this variation, the mask-predicting part of the network is not forced to match a "perfect" speech mask. Instead, the system is trained end-to-end on DOA accuracy, allowing it to learn a custom mask optimized for localization rather than human audibility.
Architecture of the mask-assisted DOA estimation network, showing the integration of mask features (Input 1) and phase features (Input 2).

The system utilizes a 6-microphone circular array, resolving sound from a full 360-degree range discretized into 72 classes (5-degree increments).

Experimental Setup

The researchers used simulated data to ensure precise ground truth labels:

  • Speech Source: Clean utterances from the TIMIT database.
  • Noise Source: Ambient noise from the CHiME 3 challenge.
  • Environment: 4,000 simulated Room Impulse Responses (RIRs) with varying sizes and reverberation times ($RT_{60}$).
  • Evaluation: Testing was conducted in both "matched" (similar to training) and "mismatched" (different room profiles) conditions to assess generalization.

Results and Discussion

The proposed methods consistently outperformed both classical signal processing and basic CNNs. At -5 dB SNR, the Multi-task model achieved 83.3% accuracy in matched conditions, significantly higher than SRP-PHAT (47.1%) and a standard CNN (78.6%). In mismatched environments, the Pseudo Multi-task model proved most robust, maintaining 65.8% accuracy at -5 dB SNR.

Visualization of the learned masks demonstrated that the Pseudo MTL network successfully identifies the target source even without explicit speech labels, learning patterns similar to Ideal Ratio Masks. This suggests that T-F masking acts as an effective attention mechanism, telling the DOA estimator which parts of the signal are trustworthy.

Visualization of the positive part of the pseudo multi-task trained mask, showing how the network learns to focus on informative frequency bands.

Conclusion

This work demonstrates that the optimal mask for localization may differ from standard speech enhancement masks. While the current model is designed for a single sound source and specific microphone geometry, it provides a robust framework for localization in noisy environments. Future work will explore multi-source scenarios and geometry-agnostic features.

Summary updated

Abstract

In the scenario with noise and reverberation, the performance of current methods for direction of arrival (DOA) estimation usually degrades significantly. Inspired by the success of time-frequency masking in speech enhancement and speech separation, this paper proposes new methods to better utilize time-frequency masking in convolution neural network to improve the robustness of localization. First a mask estimation network is developed to assist DOA estimation by either appending or multiplying the estimated masks to the original input feature. Then we further propose a multi-task learning architecture to optimize the mask and DOA estimation networks jointly, and two modes are designed and compared. Experiments show that all the proposed methods have better robustness and generalization in noisy and reverberant conditions compared to the conventional methods, and the multi-task methods have the best performance among all approaches.
BibTeX
@inproceedings{zhang2019robust,
  author = {Wangyou Zhang and Ying Zhou and Yanmin Qian},
  title = {{Robust DOA Estimation Based on Convolutional Neural Network and Time-Frequency Masking}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=VV54t17sWI},
}
Figure