Reconstructing the Dynamic Directivity of Unconstrained Speech

Camille Noufi, Dejan Markovic, Peter Dodds

Abstract

An accurate model of natural speech directivity is an important step toward achieving realistic vocal presence within a virtual communication setting. In this article, we propose a method to estimate and reconstruct the spatial energy distribution pattern of natural, unconstrained speech. We detail our method in two stages. Using recordings of speech captured by a real, static microphone array, we create a virtual array that tracks with the movement of the speaker over time. We use this egocentric virtual array to measure and encode the high-resolution directivity pattern of the speech signal as it dynamically evolves with natural speech and movement. Utilizing this encoded directivity representation, we train a machine learning model that leverages to estimate the full, dynamic directivity pattern when given a limited set of speech signals, as would be the case when speech is recorded using the microphones on a head-mounted display (HMD). We examine a variety of model architectures and training paradigms, and discuss the utility and practicality of each implementation. Our results demonstrate that neural networks can be used to regress from limited speech information to an accurate, dynamic estimation of the full directivity pattern.

Introduction

Human speech does not radiate uniformly in all directions. Instead, it forms complex, frequency-dependent patterns known as directivity. High frequencies tend to beam forward, while lower frequencies wrap around the head. For immersive AR/VR, simulating this "vocal presence" is crucial. However, existing models are often static, failing to account for head movement, posture changes, and vocal variations. The challenge lies in reconstructing these high-resolution 3D radiation patterns in real-time using only the limited microphones available on a head-mounted display (HMD).

Methodology

Researchers from Stanford and Meta proposed a two-stage framework to address this challenge:

  1. Virtual Signal Construction & Directivity Measurement: Utilizing a high-density microphone array to capture ground-truth speech, the system mathematically tracks the speaker's mouth to construct an "egocentric virtual array." This allows for the measurement of high-resolution directivity patterns, encoded via Spherical Harmonics (SH), independent of the speaker's absolute position in the room.
  2. Estimation via Deep Learning: Machine learning models are trained to predict the full SH coefficients from a limited set of 8 "proxy" signals, simulating the microphone configuration typical of an HMD.
The two-stage pipeline: (1) Virtual Signal Construction and Directivity Factor Measurement, and (2) Estimation/Reconstruction using deep learning models.

Experimental Setup and Dataset

The study utilized a custom dataset recorded in a spherical audiovisual capture chamber with a 2.74m radius, equipped with 281 first-order ambisonic microphones. The dataset consists of two minutes of continuous conversational speech from a single male speaker. A camera-based tracking system provided the necessary spatial information to maintain the egocentric coordinate system, ensuring the model learns the radiation pattern relative to the mouth rather than room position.

Visualization of frequency-dependent directivity patterns measured in the capture chamber. The rings represent the spatial energy distribution at various frequencies, showing increased complexity and narrowing at higher frequencies.

Key Results and Discussion

The evaluation compared several architectures, including Multilayer Perceptrons (MLP) and Long Short-Term Memory (LSTM) networks, alongside traditional linear baselines.

  • Model Performance: Surprisingly, the MLP outperformed the LSTM, achieving the lowest reconstruction error ($\Delta D = 0.200$). The MLP's simplicity likely made it less prone to overfitting on the small dataset, allowing it to better capture specific directivity variations.
  • Frequency Characteristics: The models successfully captured the frequency-dependent nature of speech. Below 200 Hz, radiation is nearly omnidirectional. Above 1 kHz, it becomes highly directional, often developing a "rear lobe" around 700 Hz that persists into higher frequencies.
  • Real-time Viability: The MLP achieved an inference time of just 2.1 ms, demonstrating its potential for seamless integration into real-time spatial audio engines for AR/VR.

Conclusion and Future Work

This work demonstrates that neural networks can accurately estimate dynamic speech directivity from limited microphone inputs. While successful for a single subject, future research will focus on scaling the model to diverse speakers with varying anatomy. Additionally, the team plans to validate the system with physical HMD hardware to account for acoustic shadowing and conduct perceptual listening studies to evaluate the subjective improvement in vocal presence.

Summary updated

Abstract

An accurate model of natural speech directivity is an important step toward achieving realistic vocal presence within a virtual communication setting. In this article, we propose a method to estimate and reconstruct the spatial energy distribution pattern of natural, unconstrained speech. We detail our method in two stages. Using recordings of speech captured by a real, static microphone array, we create a virtual array that tracks with the movement of the speaker over time. We use this egocentric virtual array to measure and encode the high-resolution directivity pattern of the speech signal as it dynamically evolves with natural speech and movement. Utilizing this encoded directivity representation, we train a machine learning model that leverages to estimate the full, dynamic directivity pattern when given a limited set of speech signals, as would be the case when speech is recorded using the microphones on a head-mounted display (HMD). We examine a variety of model architectures and training paradigms, and discuss the utility and practicality of each implementation. Our results demonstrate that neural networks can be used to regress from limited speech information to an accurate, dynamic estimation of the full directivity pattern.
BibTeX
@misc{noufi2022reconstructing,
  author = {Camille Noufi and Dejan Markovic and Peter Dodds},
  title = {{Reconstructing the Dynamic Directivity of Unconstrained Speech}},
  year = {2022},
  eprint = {2209.04473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.04473},
}
Figure