Spatial Audio Rendering for Real-Time Speech Translation in Virtual Meetings

Margarita Geleta, Hong Sodoma, Hannes GamperarXiv 2025 · DOI · arXiv

Introduction

Real-time speech translation (RTT) is becoming an essential tool for global collaboration. However, current implementations often suffer from a "disembodied voice" effect. In a typical virtual meeting, translated audio is delivered as a single, centered stream that lacks spatial alignment with the speaker's video window. This disconnect leads to poor speaker attribution, increased cognitive load as the brain struggles to map audio to visual cues, and informational masking where overlapping native and translated voices compete for attention. This study investigates whether spatial audio rendering—placing sound in a 3D virtual space—can improve comprehension and reduce the mental effort required in translated meetings.

Methodology

The researchers employed a "Wizard-of-Oz" (WoZ) experimental design to isolate the effects of audio rendering from the variability of AI translation quality. High-quality, pre-recorded human translations were used to simulate a perfect RTT system.

Experimental setup overview including recording environment (A), participants (B), and audio timeline (C).

The study involved 47 participants and 8 bilingual confederates who recorded 16 scripted workplace scenarios in four diverse languages: Greek, Kannada, Mandarin Chinese, and Ukrainian. These languages were selected to account for different word orders and grammar structures that affect translation latency.

Experimental Conditions

Participants were exposed to four distinct audio rendering modes to determine which provided the best user experience and comprehension:

The four audio rendering conditions: (A) Diotic, (B) Monaural, (C) Spatial, and (D) Spatial + Reverb.
  1. Diotic: Standard mono audio played identically in both ears.
  2. Monaural: Translation in one ear and original audio in the other, mimicking professional interpreter setups.
  3. Spatial: The translation is panned to match the speaker's horizontal position on the screen.
  4. Spatial + Reverb: Spatial audio enhanced with simulated room acoustics to provide a sense of physical distance.

Key Results and Insights

The study found that spatial audio significantly enhances the meeting experience across several metrics:

  • Comprehension Boost: Spatial rendering led to a 113.4% increase in the odds of a correct response on comprehension quizzes compared to the diotic baseline. Effectively, comprehension doubled.
  • Reduced Cognitive Load: Using the NASA-TLX index, researchers found that spatial cues significantly lowered mental demand. Participants felt more confident and less fatigued.
  • Stream Segregation: Spatial audio allows the brain to "spatially filter" sounds. Listeners could focus on the translation while still picking up emotional cues from the original speaker's voice.
  • Speaker Attribution: Spatial mapping made it much easier to identify "who said what," particularly during rapid turn-taking. However, the study noted that distinct voice timbres (e.g., different synthetic voices for different speakers) are still necessary when speakers have similar vocal profiles.

Conclusion

The research demonstrates that "audiovisual coherence"—aligning sound with its visual source—is critical for effective real-time translation. While professional "one-ear" monaural setups were found to be taxing for average users, spatial audio provides a natural and efficient way to manage multi-speaker translated dialogues. Future work will look into scaling these findings to larger meetings and incorporating personalized audio profiles (HRTFs) to further enhance immersion.

Summary updated

BibTeX
@misc{geleta2025spatial,
  author = {Margarita Geleta and Hong Sodoma and Hannes Gamper},
  title = {{Spatial Audio Rendering for Real-Time Speech Translation in Virtual Meetings}},
  year = {2025},
  eprint = {2511.09525},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.09525},
}
Figure