TL;DR
This paper introduces AMViT, a novel auditory cortex-inspired model that uses spectral attention to improve binaural sound localization, especially when the Head-Related Transfer Functions (HRTFs) used for training don't match those encountered during testing. AMViT achieves state-of-the-art performance in challenging HRTF mismatch and noisy conditions, making it more robust for real-world applications.
What Problem Does This Paper Address?
Precise sound localization is crucial for many applications, from virtual reality to hearing aids. Binaural sound localization (BSSL) systems, which use two ears to determine sound direction, are generally good but struggle when the acoustic properties of the listener's head and ears (represented by HRTFs) change between training and real-world use. This "HRTF mismatch" significantly degrades performance. Existing data-driven models, while effective in controlled environments, often fail to generalize to these mismatched scenarios, especially when combined with background noise. This paper aims to bridge this gap by developing a more robust BSSL system that can handle HRTF mismatches and noise effectively.
Methods
The core innovation is the Auditory Cortex-Inspired Spectral Attention Modulation (AMViT) model. It builds upon a Vision Transformer architecture, specifically adapting it for audio processing. Key aspects include:
- Spectral Attention: Inspired by how the human auditory cortex selectively focuses on specific sound frequencies, AMViT incorporates a spectral attention mechanism. This allows the model to dynamically emphasize relevant spectral features of the incoming audio.
- Top-Down and Bottom-Up Cues: The model integrates two types of information:
- Top-Down: Encoded speech representations, acting as a "prior" or expectation about the sound.
- Bottom-Up: Binaural cues like Interaural Level Difference (ILD) and Interaural Phase Difference (IPD), which are direct acoustic measurements.
- Modulation: The crucial step is how these top-down and bottom-up cues are combined. The paper explores various modulation techniques, with element-wise multiplication and an additional MLP layer proving most effective. This modulation allows the model to adapt its spectral processing based on the binaural cues, enhancing its ability to handle HRTF mismatches.
- Transformer Architecture: The model uses a multi-encoder Transformer structure, with each encoder block specialized for processing different features (spectral features, ILD, IPD). This modular design allows for efficient processing and learning of complex relationships.
Datasets
- Speech Data: TIMIT database was used for speech signals.
- HRTF Data:
- Training: 45 HRTF subjects from the CIPIC database.
- Testing: 5 HRTF subjects from the REIC database.
- Noise Data: Four types of spatially uncorrelated noise from Noisex92 and four types of non-stationary noise were used.
- Spatialization: Sound sources were spatialized to 25 azimuth angles on a horizontal plane using the HRTFs and simulated HRTFs.
Key Insights and Ideas
- Spectral Attention is Key for Mismatch: The human auditory system's ability to focus on specific frequencies is a powerful mechanism for dealing with complex acoustic environments. Applying this "spectral attention" concept to BSSL models significantly improves their robustness to HRTF variations.
- Modulation is More Than Just Combining: Simply averaging or concatenating binaural cues and spectral features isn't enough. The way these features are modulated (e.g., through multiplication) is critical for learning the complex interplay needed to overcome HRTF mismatches.
- Top-Down Priors Help Adapt: By providing the model with an internal representation of speech (top-down), it can better interpret and adapt to the incoming acoustic signals (bottom-up), even when the HRTF is unfamiliar.
- Transformer Efficiency: Despite its complexity, the Transformer architecture, when adapted for audio with specialized encoders, proves to be an efficient way to learn these intricate relationships.
Key Results
- State-of-the-Art Performance: AMViT3 (using element-wise multiplication) achieved the highest localization accuracy (53.5%) under unseen noise and HRTF conditions, outperforming previous models like DNN, CNN, and the original FAViT.
- Robustness to Mismatch: The model demonstrated significantly improved performance in HRTF mismatch scenarios, with a tolerant accuracy of 89.6% ± 1 class deviation.
- Efficiency: While AMViT3 has a larger parameter count than FAViT, it's still considered parameter-efficient compared to other large deep-learning models.
- Adjacent Azimuth Accuracy: The model shows strong accuracy in localizing sounds at adjacent azimuths, which is crucial for practical applications.
Limitations and Future Work
- HRTF Data Limitations: The study acknowledges that performance could be further improved with more HRTF data for training.
- Computational Resources: Limited computational resources constrained the amount of HRTF data used for training.
- Future Directions:
- Exploring better time-frequency speech features.
- Investigating dimensionality reduction techniques like PCA.
- Extending the model to handle reverberant conditions and elevation mismatches.
Summary updated