Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation

Jisoo Park, Seonghak Lee, Guisik Kim, Taewoo Kim, Junseok KwonarXiv 2025 · DOI · arXiv

Abstract

UniVoiceLite is an ultra-lightweight (2.3M parameters) audio-visual framework designed for simultaneous speech enhancement and separation. By leveraging an unsupervised Wasserstein Autoencoder (WAE) and visual cues such as lip motion and facial identity, the model achieves state-of-the-art signal fidelity without requiring expensive, paired noisy-clean training data. Experimental results on the GRID and DEMAND datasets demonstrate superior performance in multi-speaker scenarios and various noise environments compared to significantly larger supervised models.

Introduction

Real-world audio environments are inherently messy, often containing both background noise (requiring Speech Enhancement, SE) and overlapping talkers (requiring Speech Separation, SS). This "cocktail party problem" is traditionally addressed by large, parameter-heavy supervised models—often exceeding 70M parameters—that require perfectly paired noisy and clean audio samples for training. Such models are difficult to deploy on resource-constrained devices like hearing aids and often struggle to generalize to novel acoustic environments.

UniVoiceLite addresses these challenges by providing a unified, ultra-lightweight (2.3M parameters) framework that learns in a completely unsupervised manner, focusing on the distribution of clean speech rather than noise subtraction.

The UniVoiceLite Framework

The architecture of UniVoiceLite is built around a Wasserstein Autoencoder (WAE) that incorporates visual guidance to disentangle specific speakers from complex mixtures.

The UniVoiceLite training and inference pipeline. Training is performed on clean audio, while inference utilizes visual extractors to isolate target speech from noisy mixtures.

Visual Guidance and Unsupervised Learning

The model utilizes frozen, pre-trained visual features (e.g., from AV-HuBERT) to extract dynamic lip motion ($l_n$) and static facial identity ($f_1$). These visual priors act as anchors, allowing the model to "lock onto" a specific speaker even in crowded environments. During inference, the model uses these visual cues to reconstruct only the target speech component from a noisy input. To further refine the results, a Monte Carlo Expectation Maximization (MCEM) step with Wiener filtering is applied to suppress residual interference.

The Wasserstein Advantage

A critical innovation in UniVoiceLite is the replacement of the standard Kullback-Leibler (KL) divergence used in Variational Autoencoders (VAEs) with Wasserstein distance regularization.

Comparison between Variational Autoencoders (VAE) and Wasserstein Autoencoders (WAE). WAE prevents latent space overlap, leading to sharper audio reconstruction.

Traditional VAEs often suffer from "posterior collapse," where the model ignores latent variables, resulting in blurry, low-quality audio reconstructions. As shown in Figure 1, the WAE architecture creates a more stable and structured latent space, preventing overlap between different speakers' representations and yielding significantly higher audio reconstruction fidelity.

Experimental Results

The model was trained solely on the clean portions of the GRID corpus and evaluated using the DEMAND noise database to simulate real-world environments.

  • Signal Fidelity: In 2-speaker separation tasks, UniVoiceLite achieved a Signal-to-Distortion Ratio (SDR) of 1.46, significantly outperforming other unsupervised generative methods such as RVAE (-5.28) and AV-VAE (-3.64).
  • Intelligibility: The model consistently achieved high Short-Time Objective Intelligibility (STOI) scores across various noise types (e.g., Station, Kitchen, Metro), proving its ability to maintain speech clarity.
  • Efficiency: With only 2.3M parameters, UniVoiceLite is approximately 30x smaller than competitors like VisualVoice, yet it outperforms larger supervised models (e.g., AV-ConvTasNet at 13M params) in challenging multi-speaker scenarios.

Conclusion

UniVoiceLite demonstrates that high-performance speech enhancement and separation can be achieved with a lightweight, unsupervised model. By combining Wasserstein regularization with visual priors, the framework offers a robust solution for real-time applications on mobile and wearable devices. Future work will focus on improving perceptual quality (DNSMOS) and testing on more diverse, "in-the-wild" datasets.

Summary updated

Abstract

Speech Enhancement (SE) and Speech Separation (SS) have traditionally been treated as distinct tasks in speech processing. However, real-world audio often involves both background noise and overlapping speakers, motivating the need for a unified solution. While recent approaches have attempted to integrate SE and SS within multi-stage architectures, these approaches typically involve complex, parameter-heavy models and rely on supervised training, limiting scalability and generalization. In this work, we propose UniVoiceLite, a lightweight and unsupervised audio-visual framework that unifies SE and SS within a single model. UniVoiceLite leverages lip motion and facial identity cues to guide speech extraction and employs Wasserstein distance regularization to stabilize the latent space without requiring paired noisy-clean data. Experimental results demonstrate that UniVoiceLite achieves strong performance in both noisy and multi-speaker scenarios, combining efficiency with robust generalization. The source code is available at https://github.com/jisoo-o/UniVoiceLite.
BibTeX
@misc{park2025lightweight,
  author = {Jisoo Park and Seonghak Lee and Guisik Kim and Taewoo Kim and Junseok Kwon},
  title = {{Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation}},
  year = {2025},
  eprint = {2512.06689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.06689},
}
Figure