TL;DR
This paper tackles the challenge of tracking who is speaking in a noisy environment when the listener's head is constantly moving. They introduce a particle filtering system that uses binaural audio cues and a noisy estimate of head orientation to accurately identify speaker changes, even with significant head motion.
What Problem Does This Paper Address?
Traditional approaches to tracking sound sources, like identifying who is speaking, often assume a stationary listener with fixed microphones. However, in real-world scenarios, especially for mobile applications like robotics or wearable devices, the listener's head is in constant motion. This head movement introduces significant ambiguity: a change in the perceived direction of a sound source could be due to the source moving or the listener's head turning. This paper aims to develop a computational hearing model that can overcome these limitations by tracking speaker changes from mobile, binaural microphones, explicitly accounting for the listener's head orientation.
Methods
The core of the proposed solution is a particle filtering algorithm. This is a recursive Bayesian estimation technique that represents the probability distribution of the system's state (in this case, speaker locations and who is speaking) using a set of weighted samples (particles).
Key aspects of their method include:
- Binaural Cues: The system primarily relies on Interaural Time Differences (ITDs) extracted from the binaural audio recordings. ITDs are the differences in the arrival time of a sound at each ear, which provide crucial information about a sound source's location.
- State Space Model: They define a state space that includes the absolute spatial angle of the perceiver's head and the absolute spatial angles of each potential speaker. A crucial component is an indicator variable representing which speaker is currently active.
- Head Motion Modeling: The system explicitly models the perceiver's head movement. They assume small, Gaussian-distributed changes in head angle between time steps.
- Measurement Model: This model describes the likelihood of observing specific ITD measurements given the current state (head and speaker positions). They handle noisy ITD observations by considering the three most prominent peaks in the cross-correlogram, assuming one is the true source and the others are "clutter."
- Integration of Head Position Information: The system can optionally incorporate an estimate of the perceiver's head position. This estimate can be noisy, and the paper investigates the impact of varying levels of noise on performance.
Datasets
The study utilizes data from the CAVA database, specifically a session named "Panel Meeting 1 (P1)." This dataset was recorded using in-ear binaural microphones on a moving listener, with a head-tracking device to provide ground truth for head orientation. The dataset features five speakers seated around a table, with the listener tasked to follow the active speaker.
Key Insights and Ideas
- Head Motion is a Major Challenge: The paper clearly demonstrates that head motion significantly complicates speaker tracking, introducing ambiguity that stationary models cannot handle.
- Limited Self-Position Knowledge is Powerful: Even a noisy estimate of the perceiver's head orientation can dramatically improve the accuracy of speaker change tracking compared to having no such information at all.
- Particle Filtering is Well-Suited: The particle filtering framework proves effective for simultaneously tracking the dynamic state of both the listener's head and the active speaker.
- ITDs are Robust Enough (with caveats): While ITD extraction can be noisy, especially with head motion and reverberation, the system shows it can still be a valuable cue when processed appropriately.
Key Results
The system's performance is evaluated using the Diarization Error Rate (DER).
- When the head position is precisely known (zero noise), the system achieves a DER of around 15%. Most errors are isolated frames, suggesting potential for post-processing.
- As noise is added to the head position estimates, the DER increases. Performance degrades slowly for noise variances up to 20 degrees, then more rapidly. Beyond 50 degrees of noise, the head track information offers little advantage over no prior knowledge.
- The system also shows improved accuracy in inferring the perceiver's head orientation when even noisy self-position information is available, compared to relying solely on acoustic cues.
- The paper highlights that diarization performance deteriorates rapidly when the error in the tracked head orientation becomes comparable to the angular separation between speakers.
Limitations and Future Work
- Limitations:
- The current model assumes only one speaker is active at a time.
- Speakers are assumed to be seated at fixed, known positions and make only small movements.
- The head movement is primarily modeled as horizontal rotation.
- The system relies on a simulated, noisy head position estimate rather than a real-time sensor fusion approach.
- Future Work:
- Expanding the model to handle multiple simultaneous speakers.
- Investigating more complex head movements (e.g., pitch and roll).
- Integrating other sensory modalities (e.g., visual cues) for more robust tracking.
- Exploring more sophisticated methods for estimating speaker positions and dynamics.
Summary updated