Abstract
Data augmentation is conventionally used to inject robustness in Speaker Verification systems. Several recently organized challenges focus on handling novel acoustic environments. Deep learning based speech enhancement is a modern solution for this. Recently, a study proposed to optimize the enhancement network in the activation space of a pre-trained auxiliary network. This methodology, called deep feature loss, greatly improved over the state-of-the-art conventional x-vector based system on a children speech dataset called BabyTrain. This work analyzes various facets of that approach and asks few novel questions in that context. We first search for optimal number of auxiliary network activations, training data, and enhancement feature dimension. Experiments reveal the importance of Signal-to-Noise Ratio filtering that we employ to create a large, clean, and naturalistic corpus for enhancement network training. To counter the "mismatch" problem in enhancement, we find enhancing front-end (x-vector network) data helpful while harmful for the back-end (Probabilistic Linear Discriminant Analysis (PLDA)). Importantly, we find enhanced signals contain complementary information to original. Established by combining them in front-end, this gives ~40% relative improvement over the baseline. We also do an ablation study to remove a noise class from x-vector data augmentation and, for such systems, we establish the utility of enhancement regardless of whether it has seen that noise class itself during training. Finally, we design several dereverberation schemes to conclude ineffectiveness of deep feature loss enhancement scheme for this task.
Introduction
Automatic Speaker Verification (SV) systems—the technology used to confirm identity via voice—often struggle in real-world conditions. Background noise, babble, and reverberation distort the acoustic features necessary for accurate identification. Traditional speech enhancement focuses on improving audio for human listeners (optimizing for metrics like PESQ). However, what sounds good to a human does not always aid a machine. This paper investigates Deep Feature Loss (DFL), a task-specific enhancement method that optimizes networks using the internal activations of a pre-trained speaker recognition model rather than raw waveforms.
Methodology
The core innovation of DFL is moving the loss calculation from the time domain to the feature domain.
As shown in the architecture above, the pipeline involves:
- Enhancement Network: A Convolutional Neural Network (CNN) predicts a Time-Frequency mask, which is applied to noisy audio to produce enhanced features.
- Auxiliary Network: A pre-trained ResNet-34, trained for speaker classification, acts as a "judge."
- Deep Feature Loss: Instead of comparing enhanced audio to clean audio directly, both signals are passed through the ResNet. The loss is calculated as the difference between the internal activations (hidden layers) of the ResNet for the two signals. This forces the enhancer to prioritize features that are critical for identifying speakers.
Experimental Setup
- Training Data: The models utilized VoxCeleb1 & 2. A key finding was that filtering this data to keep only the top 50% cleanest samples (using WADA-SNR) was critical for training an effective enhancer.
- Evaluation: The system was tested on BabyTrain, a challenging dataset of daylong recordings of children in highly noisy and reverberant home environments.
- Pipeline: The study used a standard x-vector (E-TDNN) front-end for embeddings and Probabilistic Linear Discriminant Analysis (PLDA) for back-end scoring.
Key Results and Analysis
The researchers evaluated various loss configurations and training strategies to determine the most robust setup for speaker verification.
Major Findings:
- The "Aug-in-Train" Breakthrough: The most significant discovery was that enhanced signals contain information that complements the original noisy signals. Training the x-vector network on a mix of both original and enhanced data yielded a ~40% relative improvement in performance.
- Front-end vs. Back-end Mismatch: Enhancing the data used to train the x-vector network (front-end) improved results. However, enhancing the data for the PLDA (back-end) training actually degraded performance, suggesting PLDA is highly sensitive to artifacts introduced by the enhancement process.
- Layer Depth: Using activations from the first few layers of the auxiliary network generally performed better. Deeper layers may become too specialized to the specific auxiliary task, losing general acoustic information useful for broader enhancement.
- Generalization: The DFL enhancer showed robustness to unseen noise types. Even when not trained on a specific noise (like "babble"), it still improved SV performance when that noise was encountered at test time.
Conclusion
Deep Feature Loss provides a powerful mechanism for task-specific speech enhancement. While it significantly improves robustness to noise, the study concluded it is currently ineffective for dereverberation, where standard signal processing techniques like Weighted Prediction Error (WPE) still lead. Future work will explore more efficient training strategies to capture the benefits of enhanced features without doubling computational costs.
Summary updated