Abstract
Speech enhancement deep learning systems usually require large amounts of training data to operate in broad conditions or real applications. This makes the adaptability of those systems into new, low resource environments an important topic. In this work, we present the results of adapting a speech enhancement generative adversarial network by finetuning the generator with small amounts of data. We investigate the minimum requirements to obtain a stable behavior in terms of several objective metrics in two very different languages: Catalan and Korean. We also study the variability of test performance to unseen noise as a function of the amount of different types of noise available for training. Results show that adapting a pre-trained English model with 10 min of data already achieves a comparable performance to having two orders of magnitude more data. They also demonstrate the relative stability in test performance with respect to the number of training noise types.
Introduction
Deep learning models for speech enhancement—the process of removing background noise from voice recordings—typically suffer from a "data hunger" problem. Achieving robust performance across diverse real-world conditions requires massive datasets, which are often unavailable for low-resource languages or specialized environments. This research investigates whether a model can learn the fundamental task of denoising in a high-resource language (English) and effectively transfer that knowledge to others (Catalan and Korean). The study specifically examines the minimum data requirements for adaptation and the benefits of pre-training over training from scratch.
Methodology
The researchers employed the Speech Enhancement Generative Adversarial Network (SEGAN) architecture. Unlike traditional frequency-domain methods, SEGAN processes raw audio waveforms directly in the time domain.
Architecture and Transfer Strategy
The SEGAN framework consists of a Generator (G) and a Discriminator. The Generator is an encoder-decoder network with skip connections designed to produce clean audio from noisy inputs. The Discriminator's role is to distinguish between genuine clean audio and the Generator's enhanced output.
The transfer learning strategy involved taking a SEGAN model pre-trained on the English Voice Bank Corpus (30 speakers) and fine-tuning only the Generator using varying amounts of target language data. This "pre-trained" approach was benchmarked against a "from scratch" baseline where weights were initialized randomly.
Datasets
The study utilized three primary datasets:
- English: Base pre-training data from the Voice Bank Corpus.
- Korean: A dataset of 12 speakers recorded for this study.
- Catalan: Data from the FestCat project featuring 12 speakers.
- Noise: Various background environments (bus, cafe, street) mixed at different Signal-to-Noise Ratios (SNRs).
Results and Discussion
The 10-Minute Saturation Point
A key finding of the study is the high efficiency of transfer learning. Performance gains for the pre-trained model saturate remarkably quickly. While improvements were noted with as little as 24 seconds of data, 10 minutes emerged as the "sweet spot," where performance became comparable to models trained on 200 minutes of data.
Stability and Generalization
Pre-training acted as a significant stabilizer. Models trained from scratch with limited data exhibited high variance, whereas pre-trained models provided consistent results. Furthermore, the successful adaptation from English to the linguistically distinct Korean language suggests that SEGAN learns universal structural differences between human speech and stochastic noise rather than language-specific phonetics.
The variety of noise types during fine-tuning also had a minimal impact on performance against unseen noises. This indicates that the model develops a generalized representation of "noise" rather than memorizing specific acoustic profiles.
Computational Efficiency
Fine-tuning the pre-trained model required only 30 epochs, a significant reduction compared to the 86 epochs needed for the original English model training. This highlights the practical benefits of transfer learning in reducing computational costs for deploying speech enhancement systems.
Conclusion
This work demonstrates that SEGANs are highly effective at transfer learning across languages and noise conditions. By leveraging pre-trained models, high-quality speech enhancement can be achieved in low-resource settings with minimal data and computational overhead. Future research may explore simultaneous fine-tuning of the Discriminator and the application of these transfer learning principles to more modern architectures.
Summary updated