Analysis of Deep Feature Loss based Enhancement for Speaker Verification

Saurabh Kataria, Phani Sankar Nidadavolu, Jesús Villalba, Najim Dehak

Abstract

Data augmentation is conventionally used to inject robustness in Speaker Verification systems. Several recently organized challenges focus on handling novel acoustic environments. Deep learning based speech enhancement is a modern solution for this. Recently, a study proposed to optimize the enhancement network in the activation space of a pre-trained auxiliary network. This methodology, called deep feature loss, greatly improved over the state-of-the-art conventional x-vector based system on a children speech dataset called BabyTrain. This work analyzes various facets of that approach and asks few novel questions in that context. We first search for optimal number of auxiliary network activations, training data, and enhancement feature dimension. Experiments reveal the importance of Signal-to-Noise Ratio filtering that we employ to create a large, clean, and naturalistic corpus for enhancement network training. To counter the "mismatch" problem in enhancement, we find enhancing front-end (x-vector network) data helpful while harmful for the back-end (Probabilistic Linear Discriminant Analysis (PLDA)). Importantly, we find enhanced signals contain complementary information to original. Established by combining them in front-end, this gives ~40% relative improvement over the baseline. We also do an ablation study to remove a noise class from x-vector data augmentation and, for such systems, we establish the utility of enhancement regardless of whether it has seen that noise class itself during training. Finally, we design several dereverberation schemes to conclude ineffectiveness of deep feature loss enhancement scheme for this task.

Introduction

Automatic Speaker Verification (SV) systems—the technology used to confirm identity via voice—often struggle in real-world conditions. Background noise, babble, and reverberation distort the acoustic features necessary for accurate identification. Traditional speech enhancement focuses on improving audio for human listeners (optimizing for metrics like PESQ). However, what sounds good to a human does not always aid a machine. This paper investigates Deep Feature Loss (DFL), a task-specific enhancement method that optimizes networks using the internal activations of a pre-trained speaker recognition model rather than raw waveforms.

Methodology

The core innovation of DFL is moving the loss calculation from the time domain to the feature domain.

The DFL architecture: a noisy signal is enhanced by $e(\cdot)$ and compared against clean features in the activation space of a pre-trained auxiliary network $a(\cdot)$.

As shown in the architecture above, the pipeline involves:

  1. Enhancement Network: A Convolutional Neural Network (CNN) predicts a Time-Frequency mask, which is applied to noisy audio to produce enhanced features.
  2. Auxiliary Network: A pre-trained ResNet-34, trained for speaker classification, acts as a "judge."
  3. Deep Feature Loss: Instead of comparing enhanced audio to clean audio directly, both signals are passed through the ResNet. The loss is calculated as the difference between the internal activations (hidden layers) of the ResNet for the two signals. This forces the enhancer to prioritize features that are critical for identifying speakers.

Experimental Setup

  • Training Data: The models utilized VoxCeleb1 & 2. A key finding was that filtering this data to keep only the top 50% cleanest samples (using WADA-SNR) was critical for training an effective enhancer.
  • Evaluation: The system was tested on BabyTrain, a challenging dataset of daylong recordings of children in highly noisy and reverberant home environments.
  • Pipeline: The study used a standard x-vector (E-TDNN) front-end for embeddings and Probabilistic Linear Discriminant Analysis (PLDA) for back-end scoring.

Key Results and Analysis

The researchers evaluated various loss configurations and training strategies to determine the most robust setup for speaker verification.

Performance comparison (EER and minDCF) across different loss configurations and test segment lengths.

Major Findings:

  • The "Aug-in-Train" Breakthrough: The most significant discovery was that enhanced signals contain information that complements the original noisy signals. Training the x-vector network on a mix of both original and enhanced data yielded a ~40% relative improvement in performance.
  • Front-end vs. Back-end Mismatch: Enhancing the data used to train the x-vector network (front-end) improved results. However, enhancing the data for the PLDA (back-end) training actually degraded performance, suggesting PLDA is highly sensitive to artifacts introduced by the enhancement process.
  • Layer Depth: Using activations from the first few layers of the auxiliary network generally performed better. Deeper layers may become too specialized to the specific auxiliary task, losing general acoustic information useful for broader enhancement.
  • Generalization: The DFL enhancer showed robustness to unseen noise types. Even when not trained on a specific noise (like "babble"), it still improved SV performance when that noise was encountered at test time.

Conclusion

Deep Feature Loss provides a powerful mechanism for task-specific speech enhancement. While it significantly improves robustness to noise, the study concluded it is currently ineffective for dereverberation, where standard signal processing techniques like Weighted Prediction Error (WPE) still lead. Future work will explore more efficient training strategies to capture the benefits of enhanced features without doubling computational costs.

Summary updated

Abstract

Data augmentation is conventionally used to inject robustness in Speaker Verification systems. Several recently organized challenges focus on handling novel acoustic environments. Deep learning based speech enhancement is a modern solution for this. Recently, a study proposed to optimize the enhancement network in the activation space of a pre-trained auxiliary network. This methodology, called deep feature loss, greatly improved over the state-of-the-art conventional x-vector based system on a children speech dataset called BabyTrain. This work analyzes various facets of that approach and asks few novel questions in that context. We first search for optimal number of auxiliary network activations, training data, and enhancement feature dimension. Experiments reveal the importance of Signal-to-Noise Ratio filtering that we employ to create a large, clean, and naturalistic corpus for enhancement network training. To counter the "mismatch" problem in enhancement, we find enhancing front-end (x-vector network) data helpful while harmful for the back-end (Probabilistic Linear Discriminant Analysis (PLDA)). Importantly, we find enhanced signals contain complementary information to original. Established by combining them in front-end, this gives ~40% relative improvement over the baseline. We also do an ablation study to remove a noise class from x-vector data augmentation and, for such systems, we establish the utility of enhancement regardless of whether it has seen that noise class itself during training. Finally, we design several dereverberation schemes to conclude ineffectiveness of deep feature loss enhancement scheme for this task.
BibTeX
@inproceedings{kataria2020analysis,
  author = {Saurabh Kataria and Phani Sankar Nidadavolu and Jesús Villalba and Najim Dehak},
  title = {{Analysis of Deep Feature Loss based Enhancement for Speaker Verification}},
  booktitle = {Odyssey},
  year = {2020},
  url = {https://openreview.net/forum?id=YVepBYZ8Xg},
}
Figure