Abstract: The task of making speaker verification systems robust to adverse scenarios remains a challenging and active area of research. We developed an unsupervised feature enhancement approach in the log-filter bank domain with the end goal of improving speaker verification performance. We experimented with using both real speech recorded in adverse environments and degraded speech obtained by simulation to train the enhancement systems. The effectiveness of the approach was shown by testing on several real, simulated noisy, and reverberant test sets. The approach yielded significant improvements on both real and simulated sets when data augmentation was not used in the speaker verification pipeline or augmentation was used only during x-vector training. When data augmentation was used for x-vector and PLDA training, our enhancement approach yielded slight improvements.
Introduction
Speaker Verification (SV) systems—the technology used to confirm identity via voice—often struggle in "the wild." Background noise, room reverberation, and microphone distance can significantly degrade performance. While the current industry standard is to use data augmentation (artificially adding noise to training data), this approach is often too generic to account for the specific, unique distortions found in a particular real-world environment. Traditional speech enhancement typically requires "parallel data"—paired recordings of the exact same speech in both clean and noisy versions—which is nearly impossible to collect for every target scenario. This paper addresses these challenges by performing enhancement in an unsupervised way, allowing the system to learn from real-world noisy data without needing a clean ground-truth counterpart.
Methodology
The core of the proposed approach is the Unsupervised Enhancement Network (UEN), built using a Cycle-Consistent Generative Adversarial Network (CycleGAN).
- Unsupervised Learning: Unlike standard GANs, CycleGANs do not require paired examples. They learn to map features from a degraded domain (noisy/reverberant) to a clean domain by ensuring that if a signal is "cleaned" and then "re-distorted," it returns to something close to the original state.
- Feature Space: Enhancement is performed in the log Mel-filter bank (log mel-FB) space rather than on raw waveforms. This is more computationally efficient and aligns better with how modern SV systems, such as x-vectors, process audio.
- Architecture: The generator uses a fully convolutional residual network with an encoder-decoder structure, while the SV backend utilizes an Extended TDNN (ETDNN) architecture to extract speaker embeddings.
- Homogeneous Training: A key strategy explored was training the entire pipeline (the x-vector extractor and the PLDA classifier) on enhanced data, ensuring the backend is optimized for the specific artifacts introduced by the UEN.
Experimental Setup
The system was trained using a high-SNR subset of VoxCeleb1 and VoxCeleb2 as the source (clean) domain. The target (degraded) domains included:
- Simulated: VoxCeleb data convolved with simulated Room Impulse Responses (RIRs) and mixed with MUSAN noise.
- Real: Authentic acoustic environments from the AMI Meeting Corpus and Chime5 (recorded in kitchens and living rooms).
Evaluation was conducted on the Speakers in the Wild (SITW), AMI, and SRI datasets.
Results and Discussion
Robustness to Reverberation and Noise
The UEN-SV system demonstrated significant gains, particularly in reverberant conditions. On the SITW-reverb dataset, the system achieved a 21-22% relative improvement in the Minimum Decision Cost Function (minDCF) compared to a baseline SV system without enhancement.
Generalization and Real-World Performance
A key strength of the UEN is its ability to generalize to unseen noise types. Even when trained on specific MUSAN noise categories, the system maintained superior performance across various noise types and signal-to-noise ratios (SNRs), including the challenging chime3bg background noise.
Furthermore, on the real-world AMI dataset, training the UEN on actual noisy data outperformed versions trained on simulated data. This highlights the value of unsupervised learning in capturing authentic environmental distortions that simulations might miss.
Complementing State-of-the-Art Systems
While the most dramatic gains were observed in systems without data augmentation, the UEN also provided a 4.2% relative improvement on the SRI dataset when added to a system already utilizing heavy data augmentation. This suggests that unsupervised enhancement can complement existing robustness techniques in severe conditions.
Conclusion
Unsupervised feature enhancement using CycleGANs offers a powerful and flexible tool for improving speaker verification in noisy and reverberant environments. By leveraging unlabeled real-world data, the UEN can adapt to specific domains more effectively than generic data augmentation alone. Future research will focus on tighter integration with existing data augmentation pipelines and testing across even more diverse real-world scenarios.
Summary updated