Abstract
UniVoiceLite is an ultra-lightweight (2.3M parameters) audio-visual framework designed for simultaneous speech enhancement and separation. By leveraging an unsupervised Wasserstein Autoencoder (WAE) and visual cues such as lip motion and facial identity, the model achieves state-of-the-art signal fidelity without requiring expensive, paired noisy-clean training data. Experimental results on the GRID and DEMAND datasets demonstrate superior performance in multi-speaker scenarios and various noise environments compared to significantly larger supervised models.
Introduction
Real-world audio environments are inherently messy, often containing both background noise (requiring Speech Enhancement, SE) and overlapping talkers (requiring Speech Separation, SS). This "cocktail party problem" is traditionally addressed by large, parameter-heavy supervised models—often exceeding 70M parameters—that require perfectly paired noisy and clean audio samples for training. Such models are difficult to deploy on resource-constrained devices like hearing aids and often struggle to generalize to novel acoustic environments.
UniVoiceLite addresses these challenges by providing a unified, ultra-lightweight (2.3M parameters) framework that learns in a completely unsupervised manner, focusing on the distribution of clean speech rather than noise subtraction.
The UniVoiceLite Framework
The architecture of UniVoiceLite is built around a Wasserstein Autoencoder (WAE) that incorporates visual guidance to disentangle specific speakers from complex mixtures.
Visual Guidance and Unsupervised Learning
The model utilizes frozen, pre-trained visual features (e.g., from AV-HuBERT) to extract dynamic lip motion ($l_n$) and static facial identity ($f_1$). These visual priors act as anchors, allowing the model to "lock onto" a specific speaker even in crowded environments. During inference, the model uses these visual cues to reconstruct only the target speech component from a noisy input. To further refine the results, a Monte Carlo Expectation Maximization (MCEM) step with Wiener filtering is applied to suppress residual interference.
The Wasserstein Advantage
A critical innovation in UniVoiceLite is the replacement of the standard Kullback-Leibler (KL) divergence used in Variational Autoencoders (VAEs) with Wasserstein distance regularization.
Traditional VAEs often suffer from "posterior collapse," where the model ignores latent variables, resulting in blurry, low-quality audio reconstructions. As shown in Figure 1, the WAE architecture creates a more stable and structured latent space, preventing overlap between different speakers' representations and yielding significantly higher audio reconstruction fidelity.
Experimental Results
The model was trained solely on the clean portions of the GRID corpus and evaluated using the DEMAND noise database to simulate real-world environments.
- Signal Fidelity: In 2-speaker separation tasks, UniVoiceLite achieved a Signal-to-Distortion Ratio (SDR) of 1.46, significantly outperforming other unsupervised generative methods such as RVAE (-5.28) and AV-VAE (-3.64).
- Intelligibility: The model consistently achieved high Short-Time Objective Intelligibility (STOI) scores across various noise types (e.g., Station, Kitchen, Metro), proving its ability to maintain speech clarity.
- Efficiency: With only 2.3M parameters, UniVoiceLite is approximately 30x smaller than competitors like VisualVoice, yet it outperforms larger supervised models (e.g., AV-ConvTasNet at 13M params) in challenging multi-speaker scenarios.
Conclusion
UniVoiceLite demonstrates that high-performance speech enhancement and separation can be achieved with a lightweight, unsupervised model. By combining Wasserstein regularization with visual priors, the framework offers a robust solution for real-time applications on mobile and wearable devices. Future work will focus on improving perceptual quality (DNSMOS) and testing on more diverse, "in-the-wild" datasets.
Summary updated