Adversarial Separation and Adaptation Network for Far-Field Speaker Verification

Yi Lu, Man‐Wai Mak2020 · DOI

TL;DR

This paper introduces the Adversarial Separation and Adaptation Network (ADSAN), a novel deep learning architecture designed to improve far-field speaker verification by disentangling domain-specific and domain-invariant features. By incorporating a mutual information neural estimator, ADSAN effectively retains speaker discriminative information, leading to significant performance gains on challenging noisy and reverberant far-field speech.

What Problem Does This Paper Address?

Speaker verification systems, which identify individuals based on their voice, typically perform best when trained and tested on clean, close-talking microphone recordings. However, real-world scenarios often involve far-field microphones, introducing significant challenges due to environmental noise and reverberation. This "domain mismatch" between training and testing data severely degrades system performance. Existing domain adaptation techniques struggle to effectively bridge this gap, often failing to disentangle speaker-specific characteristics from environmental variations. This paper aims to address this by developing a method that can extract robust speaker embeddings even from noisy, reverberant far-field speech.

Methods

The core of the proposed solution is the Adversarial Separation and Adaptation Network (ADSAN). ADSAN builds upon the concept of Domain Separation Networks (DSNs) by employing a shared encoder to learn common features and domain-specific encoders for domain-specific features. The key innovation lies in how these features are disentangled and adapted:

  • Adversarial Learning for Separation: ADSAN uses a separation discriminator to distinguish between shared features, source-domain specific features, and target-domain specific features. Simultaneously, an adaptation discriminator tries to confuse a classifier about whether shared features originate from the source (clean) or target (noisy) domain. This adversarial process forces the shared encoder to produce domain-invariant features.
  • Speaker Discriminator: A speaker discriminator is trained on the shared features to ensure they remain speaker-discriminative, preventing the loss of crucial identity information.
  • Reconstruction Loss: A decoder attempts to reconstruct the original speech from the concatenated domain-specific and shared features. This loss backpropagates to the encoders, encouraging them to retain as much information as possible.
  • Mutual Information Neural Estimator (MINE): A critical addition is the integration of MINE. This component aims to maximize the mutual information between the input speech and the shared encoder's output. This is crucial because a flexible decoder might inadvertently discard speaker-specific information from the shared features. MINE acts as a safeguard, ensuring that the shared features retain sufficient speaker identity information, even when dealing with domain mismatch.

The ADSAN's objective is to minimize classification errors on the source domain, reconstruct original features, and enforce domain invariance and separation through adversarial training, all while maximizing mutual information.

Datasets

  • Source Domain (Clean Speech): VoxCeleb1 and VoxCeleb2 datasets were used. These were augmented to create a more challenging training set for the PLDA models.
  • Target Domain (Noisy/Reverberant Speech): The VOICES Challenge 2019 dataset was used, comprising a development set for adaptation and an evaluation set for testing.

Key Insights and Ideas

  • Disentanglement is Key: The paper strongly advocates for disentangling domain-specific and domain-invariant features as a more effective strategy for domain adaptation than simply finding a common feature space.
  • MINE as a Crucial Regularizer: The inclusion of MINE is shown to be vital. Without it, some adversarial approaches (like DANN) can suffer from "negative transfer," where performance degrades. MINE helps retain speaker discriminative information in the shared features, preventing this issue.
  • ADSAN vs. DSN: The ADSAN's approach to enforcing domain separation via classification loss (Eq. 3) is presented as a more flexible alternative to the orthogonality constraint used in traditional DSNs (Eq. 7).
  • Real-World Relevance: The focus on far-field speech and the use of the VOICES challenge dataset highlight the practical importance of this research for real-world speaker verification applications.

Key Results

The ADSAN, particularly when augmented with MINE (ADSAN+MINE), demonstrated superior performance on the challenging VOICES evaluation set compared to baseline systems (raw x-vectors, DANN, DANN+MINE, DSN, DSN+MINE).

  • On the VOICES evaluation set, ADSAN+MINE achieved the lowest minDCF (0.5989) when trained with augmented VoxCeleb data, outperforming even DSN+MINE (0.6309).
  • The paper notes that while DSN+MINE achieved the lowest EER (6.51%), ADSAN+MINE achieved the lowest minDCF, suggesting a trade-off or different strengths between the two methods.
  • Qualitative analysis (t-SNE plots, not shown) indicated that ADSAN-transformed vectors exhibited distributions closer to original vectors and smaller inter-domain distances.

Limitations and Future Work

  • MINE Convergence in DANN: The paper observes that when MINE was applied to the DANN, the mutual information loss converged to near zero, suggesting MINE might not be effective in that specific architecture. This implies that the interaction between MINE and the adversarial component is architecture-dependent.
  • Computational Cost: While not explicitly detailed, adversarial training and MINE integration generally increase computational complexity.
  • Future Work: The authors suggest that their proposed ADSAN, especially with MINE, can be further improved. The success of disentangling features and retaining speaker information opens avenues for exploring more sophisticated disentanglement techniques and applying this framework to other speech processing tasks facing domain mismatch.

Summary updated

Abstract

21st Annual Conference of the International Speech Communication Association, INTERSPEECH 2020, 25-29 October 2020, Shanghai, China
BibTeX
@misc{lu2020adversarial,
  author = {Yi Lu and Man‐Wai Mak},
  title = {{Adversarial Separation and Adaptation Network for Far-Field Speaker Verification}},
  year = {2020},
  doi = {10.21437/interspeech.2020-2372},
}
Figure