Analysis of Deep Feature Loss Based Enhancement for Speaker Verification

Saurabh Kataria, Phani Sankar Nidadavolu, Jesús Villalba, Najim Dehak

TL;DR

Applying Deep Feature Loss (DFL)—training a speech enhancement network to match activations inside a pre-trained speaker classifier—substantially improves speaker verification in noisy environments. This paper systematically explores how to configure DFL, showing that training the speaker embedding network on both raw and enhanced audio yields a massive ~40% relative error reduction, while revealing that DFL completely fails at dereverberation.

What Problem Does This Paper Address?

Standard speaker verification (SV) systems rely heavily on multi-condition data augmentation (adding noise and reverberation) to perform well in the wild. However, in extreme acoustic settings—such as BabyTrain, a dataset of day-long children's recordings in chaotic, reverberant home environments—standard x-vector systems struggle.

Explicit front-end speech enhancement can clean up the audio before verification, but conventional time-domain or spectral reconstruction losses often strip away subtle speaker characteristics. Optimizing enhancement using Deep Feature Loss (DFL)—penalizing differences in the internal representations of a pre-trained auxiliary speaker network—preserves speaker identity much better. This paper investigates how to optimize DFL pipelines, resolving critical questions around training data quality, feature dimensions, pipeline mismatches, and dereverberation.

Methods

  • Enhancement Architecture: An 8-layer Convolutional Context Aggregation Network (CAN) with dilated convolutions, Adaptive Batch Normalization, and Temporal Squeeze-and-Excitation (TSE) blocks. It predicts a log-domain time-frequency mask combined via a global skip connection.
  • Deep Feature Loss (DFL): Rather than waveform or spectrogram L1/L2 distance, the enhancement network minimizes the L1 distance between the hidden activations of an auxiliary network given clean speech versus enhanced speech.
  • Auxiliary Network: A 2D ResNet-34 with Learnable Dictionary Encoding (LDE) pooling, trained on speaker classification.
  • SV Pipeline: An Extended TDNN (E-TDNN) front-end extracting 512-dimensional x-vectors, paired with a generative Gaussian Probabilistic Linear Discriminant Analysis (PLDA) back-end.
Raw/Noisy Audio ──> [ Enhancement CAN ] ──> Enhanced Features
                           │
                 Loss = || a(Clean) - a(Enhanced) ||
                           │
Clean Audio    ──> [ Auxiliary ResNet-34 (Frozen) ]

Datasets

  • VoxCeleb 1 & 2: Concatenated into a ~2,710-hour corpus (voxcelebcat). Filtered using Waveform Amplitude Distribution Analysis (WADA-SNR) to select the cleanest 50% of utterances (~1,665 hours, vc.w) for enhancement training.
  • Noise & Impulse Responses: Noise samples from MUSAN and DEMAND; room impulse responses from the Aachen Impulse Response (AIR) database.
  • LibriSpeech: Evaluated as an alternative clean speech training source.
  • BabyTrain: The primary evaluation benchmark. Sourced from the Homebank repository, it contains day-long real-world audio of children speaking in uncontrolled, noisy, multi-speaker, and reverberant rooms.

Key Insights and Ideas

  • Cleanliness Beats Read Speech: Training the enhancer on full VoxCeleb hurts performance because it contains noisy audio. Filtering VoxCeleb by WADA-SNR works best. Switching to pristine read speech (LibriSpeech) degrades performance (EER jumped from 8.68% to 10.21%), showing that natural conversational dynamics are essential.
  • Auxiliary Embeddings Hurt: Computing DFL over all intermediate convolutional layers is optimal, but adding loss on the final speaker embedding layer degrades verification performance.
  • The PLDA Mismatch Paradox: Passing enhanced features through the x-vector front-end helps, but feeding enhanced audio into the PLDA back-end consistently hurts. PLDA is exceptionally sensitive to artificial enhancement distortions and performs best on unenhanced features.
  • Original and Enhanced Features are Complementary: Doubling the x-vector training data by feeding both original and enhanced versions of each utterance ("aug-in-train") delivered a dramatic ~40% relative boost. The two representations encode complementary acoustic cues.
  • Standard Noise Augmentation Can Harm Real-World SV: Ablating noise categories during x-vector training revealed that omitting music, babble, or background noise actually improved BabyTrain performance compared to training with all noise classes.
  • DFL Fails at Dereverberation: Whether trained standalone, in series with denoising, or multi-tasked in a two-stage network, DFL-based dereverberation failed to improve results over simple Weighted Prediction Error (WPE) pre-processing.

Key Results

Evaluated on BabyTrain across trial durations ($\ge$0s, 5s, 15s, 30s) using Equal Error Rate (EER) and minimum Detection Cost Function (minDCF):

System Configuration Mean EER (%) Mean minDCF Relative Improvement
Baseline (No Enhancement) 9.90 0.408 —
Test-time DFL Enhancement 8.68 0.357 ~12.3% EER reduction
DFL on test + front-end training 9.06 0.343 Better minDCF
Aug-in-Train (Clean + Enhanced front-end) 5.95 0.237 ~40% relative reduction
DFL Dereverberation Only 10.80 0.409 Worse than baseline

Limitations and Future Work

  • Dereverberation Ineffectiveness: Supervised DFL could not address room impulse distortion. The authors suggest unsupervised domain adaptation (such as Cycle-GANs) is likely required for reverberant environments.
  • Feature Dimension Sensitivity: Moving the enhancement network from 40-D log Mel-filterbanks to higher dimensions (80–120D or 256-D linear spectrograms) caused optimization instability and degraded verification metrics.
  • Computational Cost: The strongest setup ("aug-in-train") doubles front-end training time.
  • Model Scope: Experiments relied specifically on an E-TDNN x-vector architecture; behavior may differ on modern architectures like ECAPA-TDNN or Conformer backbones.

Summary updated

Abstract

Data augmentation is conventionally used to inject robustness in Speaker Verification systems. Several recently organized challenges focused on handling novel acoustic environments. Deep learning based speech enhancement is a modern solution for this. Recently, a study proposed to optimize the enhancement network in the activation space of a pre-trained auxiliary network.This methodology, called deep feature loss, greatly improved over the state-of-the-art conventional x-vector based system on a children speech dataset called BabyTrain. This work analyzes various facets of that approach and asks few novel questions in that context. We first search for optimal number of auxiliary network activations, training data, and enhancement feature dimension. Experiments reveal the importance of Signal-to-Noise Ratio filtering that we employ to create a large, clean, and naturalistic corpus for enhancement network training. To counter the ""mismatch"" problem in enhancement, we find enhancing front-end (x-vector network) data helpful while harmful for the back-end (Probabilistic Linear Discriminant Analysis (PLDA)).Importantly, we find enhanced signals contain complementary information to original. Established by combining them in the front-end, this gives ~40% relative improvement over the baseline. We also do an ablation study to remove a noise class from x-vector data augmentation and, for such systems, we establish the utility of enhancement regardless of whether it has seen that noise class itself during training. Finally, we design several dereverberation schemes to conclude ineffectiveness of deep feature loss enhancement scheme for this task.
BibTeX
@inproceedings{kataria2020analysis,
  author = {Saurabh Kataria and Phani Sankar Nidadavolu and Jesús Villalba and Najim Dehak},
  title = {{Analysis of Deep Feature Loss Based Enhancement for Speaker Verification}},
  booktitle = {Odyssey},
  year = {2020},
  url = {https://openreview.net/forum?id=YVepBYZ8Xg},
}

Loading related content...

Figure