Mutual Information Enhanced Training for Speaker Embedding

Youzhi Tu, Man‐Wai Mak2021 · DOI

TL;DR

This paper introduces "squeeze-DIM," a novel method to improve speaker embeddings by maximizing mutual information between frame-level audio features and the final speaker representation. By strategically "squeezing" high-dimensional frame features before mutual information estimation, they overcome a dimensionality imbalance problem, leading to more informative and discriminative speaker embeddings.

What Problem Does This Paper Address?

Modern speaker recognition systems rely on speaker embeddings – compact numerical representations of a person's voice. These embeddings are typically generated by deep neural networks (DNNs) that process audio frames, aggregate them, and produce a final segment-level embedding. While these networks are good at capturing speaker identity, there's a desire to preserve even more speaker-specific information, especially from the lower-level, more general audio features. Existing methods like Deep InfoMax (DIM) aim to maximize the mutual information (MI) between frame-level features and embeddings, but they struggle with a "dimensionality imbalance" problem: frame-level features are vastly more numerous than the embedding dimensions. This imbalance can bias the MI estimation, leading to unreliable results and even hindering performance.

Methods

The core contribution is the squeeze-DIM method. It builds upon the Deep InfoMax (DIM) framework, which uses a neural network (a "critic") to estimate the mutual information between frame-level features ($X_{conv}$) and speaker embeddings ($X_{emb}$). The key innovation is to apply a global pooling operation to the frame-level features before they are fed into the MI estimator's critic. This "squeezing" reduces the dimensionality of $X_{conv}$, mitigating the imbalance issue. The authors experimented with different global pooling methods, finding that a simple global average pooling worked well. The overall training objective combines the standard speaker classification loss with this new MI-based regularization term.

Datasets

  • Training: VoxCeleb1 development and VoxCeleb2 development data (over 2 million utterances from 7,185 speakers).
  • Evaluation:
    • VoxCeleb1 test set (clean).
  • VOiCES 2019 development and evaluation sets.

Key Insights and Ideas

  • Dimensionality Imbalance is a Real Problem: The paper clearly demonstrates that directly applying DIM to high-dimensional frame features can be detrimental due to biased MI estimation. This is a crucial practical insight for anyone using DIM-like approaches.
  • Squeezing as a Solution: The seemingly counter-intuitive step of "squeezing" (reducing dimensionality) features before MI estimation is surprisingly effective. It doesn't just reduce noise; it actively facilitates more accurate MI estimation by creating a more balanced comparison.
  • Frame-Level Choice Matters: The paper shows that the choice of which frame-level features ($X_{conv}$) to use for MI estimation is important. Features from earlier layers (closer to the raw input) seem to offer a better trade-off for MI maximization, suggesting that while information becomes more speaker-specific in later layers, the earlier layers retain more general acoustic cues that can be leveraged.
  • MI Maximization is Complementary: The proposed squeeze-DIM acts as a regularizer, enhancing the performance of standard speaker classification tasks rather than replacing them.

Key Results

The squeeze-DIM method consistently outperformed the baseline and the standard DIM approach across all evaluated datasets (VoxCeleb1, VOiCES19-dev, VOiCES19-eval). For instance, on VoxCeleb1, squeeze-DIM achieved an EER of 1.62% and minDCF of 0.167, compared to the baseline's 1.85% EER and 0.187 minDCF, and DIM's 1.94% EER and 0.189 minDCF. The paper also showed that using features from the first convolutional layer for MI estimation yielded the best results.

Limitations and Future Work

  • Information Loss from Squeezing: The authors acknowledge that the squeeze operation inherently introduces some information loss. While they show it's beneficial overall, further research could explore more sophisticated squeezing techniques that minimize this loss.
  • Choice of Pooling: While global average pooling was effective, exploring other advanced pooling methods for the "squeeze" operation could yield further improvements.
  • Source of $X_{conv}$: While earlier layers generally performed better, the paper notes that the optimal layer might depend on the specific task or dataset. Further investigation into the optimal layer selection for MI maximization could be beneficial.
  • MI Estimator Choice: The paper used InfoNCE for its stability but mentioned other estimators exist. Exploring how different MI estimators interact with squeeze-DIM could be a future direction.

Summary updated

Abstract

22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, Brno, Czechia, August 30 - September 3, 2021
BibTeX
@misc{tu2021mutual,
  author = {Youzhi Tu and Man‐Wai Mak},
  title = {{Mutual Information Enhanced Training for Speaker Embedding}},
  year = {2021},
  doi = {10.21437/interspeech.2021-1436},
}

Loading related content...

Figure