VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables

Lixing He, Yunqi Guo, Haozheng Hou, Zhenyu YanarXiv 2025 · DOI · arXiv

Abstract

VibOmni is a lightweight, multi-modal speech enhancement system designed for "earables" (like TWS earbuds) that leverages built-in Inertial Measurement Units (IMUs) to capture bone-conducted vibrations. By fusing these noise-immune vibrations with standard microphone audio, VibOmni achieves significant gains in speech clarity and word error rate reduction, even in extreme noise, while remaining efficient enough for real-time mobile execution.

Introduction

As True Wireless Stereo (TWS) earphones and AR/VR headsets become ubiquitous, our reliance on them for calls and voice assistants grows. However, their compact design creates a fundamental acoustic challenge: microphones are far from the user's mouth and close to each other, making it nearly impossible for standard beamforming to isolate the user's voice from loud ambient noise or competing speakers.

While deep learning has improved audio-only speech enhancement, these models often struggle when the environment changes (domain shift) or when noise levels are extreme. VibOmni tackles this by using the accelerometer already present in almost every modern earable to capture the subtle vibrations of the user's skull during speech, providing a noise-robust secondary signal.

Methodology

The VibOmni system employs a three-pronged approach to make bone-conduction speech enhancement scalable and effective:

1. Multi-Modal Fusion Network

The system uses a two-branch encoder-decoder architecture. One branch processes 16kHz microphone audio, while the other processes 1.6kHz vibration data from the IMU. A Dual-Path RNN (DPRNN) module fuses these features to separate speech from noise at the feature level. The model is built with causal convolutions and unidirectional RNNs to support frame-by-frame, real-time streaming.

Overview of the VibOmni system, illustrating the dual paths of bone-conducted vibration and air-conducted speech.

2. Bone Conduction Function (BCF) Augmentation

Paired audio-vibration datasets are notoriously difficult to collect. To solve this, the authors modeled the BCF—the transfer function from the vocal tract to the skull—as a Gaussian distribution in the frequency domain. This allowed them to synthetically generate vibration data from massive, existing public audio datasets (like LibriSpeech), reducing the need for manual data collection by over 70x.

3. Adaptive Inference & Continual Learning

VibOmni includes a multi-modal SNR estimator that gauges the difficulty of the current environment. In quiet settings, the system can "early-exit" or prune layers to save power. In noisy settings, it uses the estimator to identify high-quality samples for self-supervised fine-tuning on the fly.

Experimental Setup

The model was pre-trained on 1,000 hours of English speech from LibriSpeech, augmented with synthetic vibration data. Noise sources included FSD50K, Ai-shell, and DEMAND. For real-world evaluation, the team built EarSense, a prototype attached to AirPods Pro, to collect data from 32 volunteers in diverse settings like restaurants, subways, and construction sites.

Key Insights and Results

Bone-conducted vibration serves as a "golden reference" because it is physically isolated from air-conducted noise. Even with low bandwidth (typically <800Hz), it provides a reliable anchor for low-frequency speech components that are often buried in noise.

Spectrogram comparison showing how the accelerometer (Acc) captures clean speech signals even when the microphone (Mic) is overwhelmed by noise.

VibOmni achieved up to a 21% improvement in PESQ (Perceptual Evaluation of Speech Quality) and a 26% boost in Signal-to-Noise Ratio (SNR) compared to state-of-the-art baselines like FullSubNet. In "in-the-wild" tests, the system reduced the Word Error Rate (WER) by approximately 40%. Furthermore, it demonstrated 31x less latency than competing deep learning baselines, with a real-time factor of 0.12 on standard smartphones.

Discussion and Future Work

While highly effective, mass adoption faces hurdles such as Bluetooth protocol limitations for high-rate IMU streaming and the necessity of consistent earbud contact. Future work aims to move the entire pipeline onto the earable's onboard DSP (Digital Signal Processor) to eliminate smartphone dependency and further reduce latency.

Summary updated

BibTeX
@misc{he2025vibomni,
  author = {Lixing He and Yunqi Guo and Haozheng Hou and Zhenyu Yan},
  title = {{VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables}},
  year = {2025},
  eprint = {2512.02515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.02515},
}
Figure