Robust Dual-Modal Speech Keyword Spotting for XR Headsets

Zhuojiang Cai, Yuhan Ma, Feng Lu

Abstract

Keyword Spotting (KWS) is a critical component for hands-free interaction in Extended Reality (XR) environments. However, traditional vocal-only KWS systems are often compromised by environmental noise, social privacy concerns, and external interference. This paper presents a robust dual-modal KWS system that fuses traditional vocal audio with ultrasonic echoic sensing of mouth movements. By integrating a low-power "vocal-echoic" sensor suite into an XR headset, the system achieves high accuracy in noisy environments, enables "silent speech" commands, and effectively rejects interference from nearby speakers.

Introduction

The ubiquity of XR headsets necessitates reliable hands-free control, typically achieved through voice-activated keywords like "Hey Cortana." Despite their convenience, current KWS technologies face significant challenges:

  1. Acoustic Noise: High-decibel environments (e.g., subways, busy streets) mask user commands.
  2. Social Constraints: Users may prefer not to speak aloud in quiet public spaces like libraries.
  3. Interference: Commands from bystanders can lead to false triggers.
Overview of the proposed dual-modal KWS system and its application in typical, noisy, and silent environments.

To address these, we propose a system that supplements vocal audio with ultrasonic sensing, creating a robust interface capable of operating across diverse scenarios.

Methods

The proposed system is implemented on a Microsoft HoloLens 2, augmented with a custom ESP32-based hardware module.

Hardware and Sensing Modalities

The system utilizes two distinct modalities:

  • Vocal Modality: Standard microphones capture audio below 10 kHz. Features are extracted using Mel-frequency cepstral coefficients (MFCCs) and processed via a Broadcast Residual CNN.
  • Echoic Modality: This novel component uses ultrasonic speakers to emit Frequency-Modulated Continuous Wave (FMCW) chirps (17–23.5 kHz). These waves reflect off the user's mouth, creating an "Echo Profile" that tracks physical movement. This data is analyzed by a lightweight ResNet-18 variant.
Detailed view of the vocal-echoic sensing mechanism.

Signal Fusion

Two fusion strategies were evaluated to combine the vocal and echoic streams:

  1. Reliability-based (RB) Fusion: A heuristic approach that weights modalities based on environmental noise levels.
  2. MLP-based Fusion: A Multi-Layer Perceptron that learns optimal weights for merging prediction vectors from both modalities.

Experimental Results

The system was trained using the Google Speech Commands dataset for vocal audio and a proprietary dataset of 10 command words recorded from 15 participants for the echoic modality.

Robustness and Accuracy

In high-noise conditions (-10dB SNR), the MLP-based fusion significantly outperformed single-modality baselines, reducing the Word Error Rate (WER) by approximately 16%. Furthermore, the system successfully identified "silent speech" (mouth movements without vocalization), a feat impossible for traditional KWS.

Efficiency for Edge Deployment

To ensure feasibility for battery-powered XR devices, the echoic model was optimized using depth-wise separable convolutions. The resulting model (ResNet-18-1/4-DS) contains only 109.9K parameters—a 100x reduction from the baseline—while maintaining a low WER of 5.97%.

Conclusion and Future Work

The dual-modal vocal-echoic approach provides a robust solution for KWS in XR, overcoming the limitations of purely acoustic systems. While physical activity (e.g., walking) currently introduces motion artifacts that increase WER, future iterations will incorporate activity-aware filtering. Future work also aims to integrate the sensors directly into the headset frames for a more seamless form factor.

Summary updated

BibTeX
@article{cai2024robust,
  author = {Zhuojiang Cai and Yuhan Ma and Feng Lu},
  title = {{Robust Dual-Modal Speech Keyword Spotting for XR Headsets}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2024},
  doi = {10.1109/tvcg.2024.3372092},
}
Figure