Improving Embedding Extraction for Speaker Verification with Ladder Network

Fei Tao, Gökhan TürarXiv 2020 · OpenReview

TL;DR

This paper introduces a "ladder network" approach to improve speaker verification by enhancing how deep neural networks extract voice embeddings. By combining supervised learning (for speaker identification) with unsupervised learning (for reconstructing input signals), the method helps models retain more generalizable voice characteristics, leading to better performance without needing more data or parameters.

What Problem Does This Paper Address?

Speaker verification (SV) is the task of confirming if a person is who they claim to be, based on their voice. Modern SV systems rely on deep neural networks (DNNs) to create "embeddings" – compact representations of a speaker's voice that capture unique characteristics. While much research focuses on making these embeddings more discriminative (better at telling speakers apart), there's less focus on improving their generalization. This is crucial because SV is an "open set" problem: the system might encounter voices it wasn't explicitly trained on (e.g., different recording conditions, microphones, or background noise). Existing supervised learning methods in DNNs tend to discard information deemed "useless" for the specific training task, which can hurt performance on unseen data. This paper aims to address this by finding a way to retain more of the original voice information that might be useful for generalization, without sacrificing discriminative power.

Methods

The core innovation is the application of the Ladder Network framework to speaker embedding extraction. A ladder network combines a standard supervised classification network with a denoising autoencoder (DAE) structure.

Here's how it works:

  • Classification Network: This is the standard DNN used for speaker verification (e.g., d-vector or x-vector architectures). It's trained using supervised learning (e.g., to predict speaker identity).
  • Denoising Autoencoder (DAE): This part is integrated within the classification network, operating at each layer.
  • Encoder: Shares weights with the classification network. It takes the output of a layer and adds Gaussian noise.
  • Decoder: Tries to reconstruct the clean version of the noisy layer output. It uses parameters learned from the classification network's layers and also receives information from the layer above it in the decoder path.
  • Combined Training: The network is trained simultaneously on two objectives:
  • Supervised Loss: The standard classification loss (e.g., cross-entropy for speaker identification). This encourages discriminative embeddings.
  • Unsupervised Denoising Loss: The reconstruction error from the DAE. This encourages the network to preserve useful information in the embeddings, acting as a form of regularization and improving generalization.

The authors apply this ladder network concept to two state-of-the-art SV embedding extractors:

  • d-vector: A simpler, fully connected network suitable for short audio segments.
  • x-vector: A more complex Time Delay Neural Network (TDNN) designed for longer audio segments, incorporating statistics pooling. For the x-vector, the ladder network is applied only to the frame-level layers before the statistics pooling, as reconstructing segment-level statistics is not feasible.

Datasets

The experiments were conducted using the VoxCeleb corpus (Phase 1 and 2). This is a large-scale, publicly available dataset derived from YouTube videos, featuring natural speech from thousands of speakers in diverse, uncontrolled environments. The authors used data from Phase 2 and the "develop" set of Phase 1 for training, and the "test" set from Phase 1 (containing 40 speakers) for evaluation. No new benchmarks were introduced; the focus was on evaluating the proposed method on existing, well-established datasets.

Key Insights and Ideas

  • Balancing Discriminability and Generalization: The core idea is that supervised learning alone can over-specialize, discarding potentially useful information. The ladder network's unsupervised denoising task acts as a regularizer, forcing the network to retain more generalizable features.
  • Layer-wise Denoising: Applying the denoising process at each layer of the network, rather than just as a pre-training step, is key. This ensures that useful information is preserved throughout the embedding extraction pipeline.
  • Ladder Network vs. Multi-Task Learning: The paper compares the ladder network to a more traditional multi-task learning approach (speaker identification + direct input reconstruction). The ladder network outperformed the multi-task approach, suggesting that the way the unsupervised task is integrated (forcing each layer to learn reconstruction) is more effective than simply adding a secondary reconstruction objective.
  • Regularization Effect: The ladder network's training process appears to prevent overfitting, leading to more stable performance curves and better generalization compared to the baseline x-vector.

Key Results

The proposed ladder network approach, termed "d-ladder" and "x-ladder," showed improvements over their respective baselines ("d-vector" and "x-vector"):

  • d-ladder: Achieved a relative improvement of 10% in Equal Error Rate (EER) compared to the d-vector baseline.
  • x-ladder: Achieved a relative improvement of 2% in EER compared to the x-vector baseline.
  • These improvements were achieved without adding extra parameters or using data augmentation.
  • The x-vector framework (and by extension, x-ladder) significantly outperformed the d-vector framework, attributed to its ability to model longer temporal contexts.
  • The "x-ladder" demonstrated more stable training and less overfitting compared to the "x-vector" baseline, as visualized by its EER performance over epochs.

Limitations and Future Work

  • Training Time: The ladder network approach resulted in longer training times for both d-vector (45 hrs vs. 40 hrs) and x-vector (180 hrs vs. 106 hrs) systems.
  • Limited Improvement for X-vector: The relative improvement for the x-vector was smaller (2%) compared to the d-vector (10%). The authors attribute this to the x-vector already being a highly developed, state-of-the-art system with less room for improvement.
  • Future Work: The authors suggest further investigation into how much more improvement can be achieved by continuing the training process for the ladder network systems. They also imply that the ladder network's ability to regularize and generalize could be explored in other speech processing tasks.

Summary updated

Abstract

Speaker verification is an established yet challenging task in speech processing and a very vibrant research area. Recent speaker verification (SV) systems rely on deep neural networks to extract high-level embeddings which are able to characterize the users' voices. Most of the studies have investigated on improving the discriminability of the networks to extract better embeddings for performances improvement. However, only few research focus on improving the generalization. In this paper, we propose to apply the ladder network framework in the SV systems, which combines the supervised and unsupervised learning fashions. The ladder network can make the system to have better high-level embedding by balancing the trade-off to keep/discard as much useful/useless information as possible. We evaluated the framework on two state-of-the-art SV systems, d-vector and x-vector, which can be used for different use cases. The experiments showed that the proposed approach relatively improved the performance by 10% at most without adding parameters and augmented data.
BibTeX
@misc{tao2020improving,
  author = {Fei Tao and Gökhan Tür},
  title = {{Improving Embedding Extraction for Speaker Verification with Ladder Network}},
  year = {2020},
  eprint = {2003.09125},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.09125},
}
Figure