Binaural sound source localization using a hybrid time and frequency domain model

Gil Geva, Olivier Warusfel, Shlomo Dubnov, Tammuz Dubnov, Amir Amedi, Yacov Hel-Or

Abstract

This paper introduces a novel approach to sound source localization using head-related transfer function (HRTF) characteristics, enabling precise full-sphere localization from raw binaural data. While previous research primarily utilized extensive microphone arrays or focused on the frontal plane, these methods often encountered limitations in accuracy and robustness when scaled down to smaller arrays. Our model proposes a hybrid Deep Learning (DL) architecture that processes both time and frequency domain data simultaneously. The performance of this proposed model significantly surpasses current state-of-the-art results, achieving an average angular error of 0.24° and an average Euclidean distance of 0.01 meters. This level of precision is critical for applications in robotics, virtual reality, and the development of advanced cochlear implants (CI).

Introduction

Sound localization—the ability to determine the origin of a sound in three-dimensional space—is a task humans perform naturally using just two ears. However, replicating this capability in technology has historically required large, impractical microphone arrays. For users of wearable devices like hearing aids or cochlear implants, these bulky arrays are not a viable solution.

Furthermore, most existing binaural (two-microphone) research is limited to the "frontal plane," covering only the 180 degrees in front of the listener. When attempting full-sphere 360-degree localization with small arrays, accuracy typically plummets due to the "cone of confusion," where sounds from the front and back appear identical to the sensors. This paper presents a significant leap forward, moving from traditional frontal arcs to precise, full-sphere 3D tracking using a hybrid deep learning architecture.

Methodology: The Hybrid Approach

The core innovation of this research is the Hybrid Time and Frequency Domain Model. The authors argue that neither time nor frequency data is sufficient on its own for full-sphere localization. Instead, their end-to-end deep learning approach leverages two distinct data streams:

  1. Raw Waveforms (Time Domain): These preserve fine-grained temporal details, such as Interaural Time Differences (ITD)—the micro-delays between a sound hitting one ear versus the other.
  2. Spectrograms (Frequency Domain): Generated via Short-Time Fourier Transform (STFT), these highlight Interaural Level Differences (ILD) and spectral cues caused by the physical shape of the outer ear (the pinna).

Architecture and Loss Function

The model utilizes a dual-stream branch system. One branch processes raw audio through convolutional layers, while the parallel branch processes spectrograms. These features are concatenated and fed into a deep neural network (DNN) with ReLU activation.

To improve stability, the researchers moved away from standard polar coordinates, which often struggle with convergence in neural networks. Instead, they utilized Cartesian coordinates (x, y, z). Their custom loss function combines Euclidean distance and angular error, prioritizing directional accuracy over simple proximity.

Experimental Setup

To train and validate the model, the researchers combined high-fidelity spatial recordings with musical data:

  • HRIR Collection: Head-Related Impulse Responses (HRIRs) were recorded at the IRCAM studio in Paris using a Neumann KU 100 dummy head. Twenty-four strategically positioned speakers were used to map the full 3D sphere.
  • Data Synthesis: The team utilized the MUSDB18 dataset, convolving song samples with the HRIRs to simulate audio originating from specific 3D locations around the dummy head.
  • Evaluation: A rigorous 30% training / 15% testing split was implemented, ensuring that specific songs used in the test set were never seen by the model during training to prevent data contamination.

Key Results and Insights

The hybrid model's performance represents an order-of-magnitude improvement over previous benchmarks. By successfully learning to interpret the subtle spectral "fingerprints" left by the pinna, the model effectively resolved the front-back ambiguity that plagues traditional systems.

Metric Previous State-of-the-Art Proposed Hybrid Model
Average Angular Error 19.07° 0.24°
Average Euclidean Distance 1.08 m 0.01 m

Beyond the raw metrics, the model demonstrated remarkable horizontal robustness. While benchmark models typically degrade in accuracy as the horizontal angle increases, the hybrid model remained consistently accurate across the entire 3D sphere.

Discussion and Future Work

While these results set a new standard for binaural localization, the authors identify several areas for future development. Currently, the model is trained on a specific dummy head. Because human ear shapes are unique, the next step is making the model "head-agnostic" so it can function for diverse users without custom retraining.

Additionally, while the current study was conducted in a controlled environment, future iterations will need to incorporate noise cancellation and source separation to handle complex "cocktail party" scenarios. Interestingly, the authors suggest that by further optimizing pinna design, it may eventually be possible to achieve 3D localization using only a single microphone, potentially revolutionizing the size and cost of wearable acoustic technology.

Summary updated

Abstract

This paper introduces a new approach to sound source localization using head-related transfer function (HRTF) characteristics, which enable precise full-sphere localization from raw data. While previous research focused primarily on using extensive microphone arrays in the frontal plane, this arrangement often encountered limitations in accuracy and robustness when dealing with smaller microphone arrays. Our model proposes using both time and frequency domain for sound source localization while utilizing Deep Learning (DL) approach. The performance of our proposed model, surpasses the current state-of-the-art results. Specifically, it boasts an average angular error of $0.24 degrees and an average Euclidean distance of 0.01 meters, while the known state-of-the-art gives average angular error of 19.07 degrees and average Euclidean distance of 1.08 meters. This level of accuracy is of paramount importance for a wide range of applications, including robotics, virtual reality, and aiding individuals with cochlear implants (CI).
BibTeX
@inproceedings{geva2024binaural,
  author = {Gil Geva and Olivier Warusfel and Shlomo Dubnov and Tammuz Dubnov and Amir Amedi and Yacov Hel-Or},
  title = {{Binaural sound source localization using a hybrid time and frequency domain model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=CAwDt37yWA},
}
Figure