Introduction
Speech Enhancement (SE) systems have seen significant advancements through deep learning, yet they often struggle with a "generalization gap." While a model may perform well on average, it often fails to account for the unique vocal characteristics—such as pitch and cadence—of individual users. Traditional fine-tuning requires substantial data and compute, which is impractical for real-time applications.
OSSEM (One-Shot Speaker-adaptive Speech Enhancement) addresses this by enabling a system to optimize itself for a new speaker using just a single utterance. This "one-shot" adaptation allows for personalized, high-quality audio processing in real-time, causal environments, making it suitable for mobile devices and hearing aids.
System Architecture
The OSSEM framework decouples the enhancement task into two specialized components:
- Transformer SE Network: A robust backbone designed for general-purpose speech enhancement, utilizing multi-head attention to capture long-range dependencies in audio signals.
- Speaker-Specific Masking (SSM) Network: A lightweight module that processes speaker embeddings (voice fingerprints) to generate a mask. This mask is applied to noisy input features before they enter the main SE network.
Research indicates that "Pre" masking—applying the speaker-specific adjustments at the very beginning of the pipeline—yields the most significant performance gains. This early intervention allows the subsequent Transformer layers to work with features already tailored to the target speaker's voice. To ensure stability during rapid adaptation, the authors also implemented a critical feature re-scaling technique to balance energy ratios between clean and noisy features.
Meta-Learning for Rapid Adaptation
To achieve effective one-shot adaptation, OSSEM employs a meta-learning strategy inspired by Model-Agnostic Meta-Learning (MAML). The goal is not just to clean audio, but to learn a set of parameters that can be quickly adapted to any new speaker with minimal gradient updates.
The training process involves two nested loops:
- Inner Loop (Speaker Adaptation): The model attempts to adapt to a specific speaker using a single "enrollment" sample. Crucially, only the SSM network is updated during this phase (the "Speaker-Inner-Loop"), ensuring stability and preventing the model from forgetting general speech patterns.
- Outer Loop (Meta-Update): The system updates its global parameters based on the performance of the adapted model on a separate "query" set. This ensures the base model remains highly "trainable" for future unseen speakers.
Experimental Results
Evaluated on the Voice Bank-DEMAND dataset using ECAPA-TDNN speaker embeddings, OSSEM demonstrates superior performance compared to existing causal SE models across standard metrics like PESQ (quality) and STOI (intelligibility).
- Quality and Intelligibility: OSSEM achieved a PESQ score of 2.90, outperforming the baseline Transformer (2.69) and Conv-TasNet (2.53).
- Efficiency: With a model size of 38MB, OSSEM is nearly half the size of the popular DEMUCS model (73MB) while maintaining full causality, making it ideal for real-time hardware deployment.
- Adaptation Value: The adaptive OSSEM consistently outperformed non-adaptive speaker-aware versions, confirming that the meta-learning process successfully captures speaker-specific nuances that static embeddings alone cannot. Visualizations further reveal that the SSM network effectively clusters speakers by spectral characteristics, creating custom filters for different voice types.
Conclusion
OSSEM represents a significant step toward truly personalized audio technology. By combining a Transformer architecture with a meta-learned masking network, it bridges the generalization gap with minimal data. Future work will explore the application of this framework to more complex tasks such as multi-speaker separation and target speaker extraction in highly adverse acoustic environments.
Summary updated