Unsupervised Feature Enhancement for Speaker Verification

Phani Sankar Nidadavolu, Saurabh Kataria, Jesús Villalba, L. Paola García-Perera, Najim Dehak

TL;DR

Researchers from Johns Hopkins University have developed an unsupervised feature enhancement system using CycleGANs to make speaker verification more robust in noisy and reverberant environments. By training on unlabeled, non-parallel data, the system successfully "cleans" speech features, providing up to a 22% improvement in simulated conditions and a 4.2% boost to state-of-the-art systems on challenging real-world datasets.

What Problem Does This Paper Address?

Speaker Verification (SV) systems—the tech that confirms your identity via your voice—often struggle in "the wild." Background noise, room echoes (reverberation), and distance from the microphone can significantly degrade performance.

While the current standard is to use data augmentation (artificially adding noise to training data), this approach is generic. It doesn't account for the specific, unique distortions found in a particular real-world environment. Traditional speech enhancement often requires "parallel data"—recordings of the exact same speech in both clean and noisy versions—which is nearly impossible to collect for every target scenario. This paper tackles the challenge of performing enhancement in an unsupervised way, allowing the system to learn from real-world noisy data without needing a clean "ground truth" counterpart.

Methods

The core of the approach is the Unsupervised Enhancement Network (UEN), built using a Cycle-Consistent Generative Adversarial Network (CycleGAN).

  • Unsupervised Learning: Unlike standard GANs, CycleGANs don't need paired examples. They learn to map features from one domain (noisy/reverberant) to another (clean) by ensuring that if you "clean" a signal and then "re-distort" it, you get back something close to the original.
  • Feature Space: The enhancement happens in the log Mel-filter bank (log mel-FB) space rather than on raw waveforms. This is more computationally efficient and aligns better with how modern SV systems (like x-vectors) process audio.
  • Architecture: The generator uses a full-convolutional residual network with an encoder-decoder structure, while the SV backend uses an Extended TDNN (ETDNN) architecture to extract speaker embeddings (x-vectors).
  • Homogeneous Training: A key strategy explored was training the entire pipeline (the x-vector extractor and the PLDA classifier) on "enhanced" augmented data, ensuring the backend is familiar with the specific artifacts introduced by the UEN.

Datasets

  • Source (Clean) Domain: A high-SNR subset of VoxCeleb1 and VoxCeleb2, filtered using the WADASNR algorithm to ensure high quality.
  • Target (Degraded) Domain:
    • Simulated: VoxCeleb data convolved with simulated Room Impulse Responses (RIRs) and mixed with MUSAN noise.
  • Real: The AMI Meeting Corpus and Chime5 (recorded in kitchens/living rooms), providing authentic, uncontrolled acoustic environments.
  • Evaluation Sets: Speakers in the Wild (SITW), AMI, and SRI (a challenging indoor controlled setting with severe background noise).

Key Insights and Ideas

  • Domain Knowledge without Labels: The UEN can incorporate knowledge of a specific adverse domain (like a specific office or kitchen) simply by feeding it unlabeled recordings from that domain.
  • Simultaneous Denoising and Dereverberation: The researchers successfully developed a unified network that handles both types of distortion at once, rather than treating them as separate sequential steps.
  • Generalization: Even when the UEN was trained on specific noise types (like MUSAN), it showed a surprising ability to generalize to unseen noise conditions (like the "chime3bg" noise).
  • The "Homogeneous" Advantage: The best results came when the speaker verification backend was "aware" of the enhancement. Training the x-vector system on data that had already passed through the UEN yielded the most significant gains on the toughest datasets.

Key Results

  • Simulated Reverb: On the SITW-reverb set, the UEN-SV system achieved a 21-22% relative improvement in Minimum Decision Cost Function (minDCF) compared to a baseline SV system.
  • Real-World Performance: On the AMI dataset, using UEN trained on real data outperformed versions trained on simulated data, highlighting the value of unsupervised learning from real-world samples.
  • Boosting the State-of-the-Art: When added to a system already using heavy data augmentation, the UEN provided a 4.2% relative improvement on the SRI dataset, proving it complements existing robustness techniques in severe conditions.

Limitations and Future Work

The authors note that the benefits of UEN are most pronounced when the baseline system doesn't use data augmentation. When the backend is already heavily augmented, the UEN's gains are more modest.

Future research directions include:

  • Exploring how to better integrate the UEN with existing data augmentation pipelines.
  • Testing the system on even more diverse real-world domains.
  • Investigating if the enhancement introduces artifacts that might affect other speech tasks, like diarization or speech recognition.

Summary updated

Abstract

The task of making speaker verification systems robust to adverse scenarios remains a challenging and an active area of research. We developed an unsupervised feature enhancement approach in log-filter bank space with the end goal of improving speaker verification performance. We experimented with using both real speech recorded in adverse environments and degraded speech obtained by simulation to train the enhancement systems. The effectiveness of this approach was shown by testing on several real, simulated noisy, and reverberant test sets. The approach yielded significant improvements on both real and simulated sets when data augmentation was not used in speaker verification pipeline. We also experimented with training the x-vector and PLDA systems with enhanced augmented features instead of augmented features and observed better performance on real test conditions (4.2% relative improvement in minDCF on SRI).
BibTeX
@inproceedings{nidadavolu2020unsupervised,
  author = {Phani Sankar Nidadavolu and Saurabh Kataria and Jesús Villalba and L. Paola García-Perera and Najim Dehak},
  title = {{Unsupervised Feature Enhancement for Speaker Verification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=DWiRAsbK3F},
}

Loading related content...

Figure