Abstract
Researchers from the University of Maryland have developed a system that uses binaural (two-microphone) recordings to simultaneously track both a speaker's location and their head orientation. By leveraging a deep learning model that accounts for near-field acoustic filtering, the system achieves high precision—within 2.5° for listener orientation—opening doors for more intuitive AR/VR and human-robot interactions.
Introduction
In the evolving field of "earables" (smart headphones and hearing aids), determining the source of a sound is only part of the challenge. To truly understand social context—such as whether someone is talking directly to you or to another person—a device must estimate both the speaker’s direction (angle relative to the listener) and their orientation (the direction their head is turned).
This task is notoriously difficult due to the "acoustic fingerprint" created by the human head (Head-Related Transfer Function, or HRTF) and the fact that human speech is highly directional, with high frequencies beaming forward while low frequencies wrap around the head. This paper addresses these challenges using just two synchronized microphones, common in modern earbuds.
Methodology
Geometric Modeling
The authors propose a joint estimation framework that treats the speaker-listener relationship as a complex geometric puzzle. Unlike most audio research that assumes far-field sound waves, this study focuses on "near-field" distances (0.5m to 1.5m) typical of social interactions. The model accounts for the curvature of sound waves and the parallax effect created by the distance between a listener's ears.
Feature Engineering and Model Architecture
Rather than using raw audio, the system extracts specific physical cues, including Interaural Level and Time Differences (ILD & ITD). These features are split into high and low-frequency bands; high frequencies carry more orientation information, while low frequencies are better for direction. A novel "inverse convolution" feature is also used to calculate high-to-low energy ratios, emphasizing the speaker's facing direction.
These features are processed by an 8-layer 1-D Convolutional Neural Network (CNN) that regresses both target angles simultaneously. The authors found that joint learning helps the network disentangle overlapping acoustic cues more effectively than separate models.
Dataset Synthesis
To train the model, the researchers synthesized a massive dataset of 560,000 recordings. They combined real-world HRTF measurements from the RIEC Database, human Voice Directivity Patterns (VDP), and over 100,000 utterances from the Speech Commands Dataset, applying near-field corrections to create a realistic training environment.
Key Results and Insights
The system was evaluated on its ability to predict angles and classify social states, yielding several key insights:
- High Precision: The system achieved a 90th percentile error of only 2.5° for listener orientation and 12.5° for speaker orientation.
- Near-field Advantage: Including near-field corrections improved the 80th percentile error for direction from 8° to just 1.3°.
- Social Classification: The model could determine if two people were "facing & facing" (engaged in conversation) with over 90% accuracy.
- Voiced Speech Reliability: The system performs significantly better on "voiced" segments (like vowels) because they have a stronger harmonic structure and higher directivity compared to unvoiced sounds.
Conclusion and Future Work
While the current results are based on synthetic data, they demonstrate the feasibility of high-precision spatial sensing in consumer wearables. Future work will focus on dynamic tracking of moving speakers and improving robustness in reverberant real-world environments. The ultimate goal is to implement these 1-D CNN models on low-power processors suitable for integration into commercial smart headphones.
Summary updated