Analysis of Complementary Information Sources in the Speaker Embeddings Framework

Mahesh Kumar Nandwana, Mitchell McLaren, Diego Castan, Julien van Hout, Aaron LawsonInterspeech 2018

TL;DR

This paper delves into how to improve speaker recognition systems by analyzing and combining different information sources within deep neural network (DNN) speaker embeddings. They found that different layers within a DNN embedding extractor capture complementary information, and fusing these layers, along with different acoustic features, can significantly boost performance.

What Problem Does This Paper Address?

Speaker recognition, the task of identifying who is speaking, has seen a revolution with the advent of deep neural networks (DNNs) that generate "speaker embeddings" – fixed-size vector representations of a speaker's voice. While these DNN-based embeddings have achieved state-of-the-art results, there's a need to understand how they work and how to extract the most information from them. Specifically, this paper addresses the question of how to best leverage complementary information from different parts of the DNN embedding extraction process and from different acoustic features to improve the robustness and accuracy of speaker recognition systems. This is crucial for real-world applications where performance needs to be balanced with computational efficiency.

Methods

The core of this work lies in exploring different ways to combine information sources within a DNN-based speaker recognition framework. The authors use a system composed of:

  • Speech Activity Detection (SAD): A DNN-based module to identify speech segments.
  • Speaker Embedding Extractor: A feed-forward DNN that takes speech segments and outputs a fixed-dimensional speaker embedding. This DNN has multiple hidden layers, and the authors investigate extracting embeddings from the first and second segment-level hidden layers, noting their complementary nature.
  • Probabilistic Linear Discriminant Analysis (PLDA) Classifier: A standard classifier used to compare speaker embeddings and generate similarity scores.

The key methodological contribution is the investigation of three fusion strategies:

  1. Score-Level Fusion: Combining the output scores from different systems or components. They explored two methods:
  2. Score Averaging: A simple, equally weighted sum of scores.
  3. Logistic Regression: A more sophisticated method that learns optimal weights for combining scores, often used for calibration as well.
  4. Embeddings-Level Fusion: Combining the embedding vectors themselves. This involves reducing the dimensionality of individual embeddings (using LDA), concatenating them, and then applying another LDA transform to get a fused embedding.
  5. Feature-Level Fusion: Concatenating raw acoustic features before they are fed into the embedding extractor DNN. This is an early-stage fusion approach.

They also benchmarked three different front-end acoustic features: Mel-frequency cepstral coefficients (MFCC), power normalized cepstral coefficients (PNCC), and perceptual linear prediction (PLP).

Datasets

The experiments were conducted on two datasets:

  • NIST SRE 2016: A standard benchmark for speaker recognition, featuring various conditions like domain/channel mismatches, language variations, and different recording types.
  • Speakers in the Wild (SITW): A more recent dataset containing naturally occurring speech with various noises, reverberation, and channel variability.

The development sets of these datasets were used for training fusion parameters (specifically for logistic regression).

Key Insights and Ideas

  • Complementarity of DNN Layers: The first and second segment-level hidden layers of the DNN embedding extractor capture distinct, complementary information. This means fusing embeddings from these layers can improve performance.
  • Feature Robustness: PNCC features generally outperformed MFCC and PLP for speaker recognition across different conditions, especially on the SRE 2016 dataset. However, MFCC with the second embedding layer showed strong performance on SITW. This highlights that no single feature is universally best.
  • Fusion Effectiveness: Score-level and embeddings-level fusion are effective strategies for combining complementary information, leading to significant performance gains.
  • Feature-Level Fusion Challenges: Concatenating raw acoustic features (feature-level fusion) did not yield significant improvements and sometimes degraded performance. The authors suggest this might be because the features are highly correlated and require dimensionality reduction before fusion.

Key Results

  • Single System Complementarity:
    • Applying score-level or embeddings-level fusion between the first and second embedding layers resulted in relative improvements of 17% EER on NIST SRE 2016 and 10% EER on SITW over the baseline system.
  • Logistic regression fusion was particularly effective when one feature (PLP) was less performant, suggesting it can better adapt weights than simple averaging.
  • Front-End Feature Performance:
    • PNCC features provided a relative improvement of 9.6% for SRE 16 compared to MFCC.
  • MFCC with the second layer embedding improved performance by over 6% for SITW.
  • PLP features generally performed the worst.
  • Multiple Feature Combination: While combining multiple acoustic features (MFCC+PNCC, MFCC+PLP, PNCC+PLP) at the score or embedding level showed some improvements, they did not outperform the single-system complementarity from different layers of the same DNN.

Limitations and Future Work

  • Feature-Level Fusion: The authors acknowledge that feature-level fusion, as implemented, did not yield benefits and suggest that dimensionality reduction techniques (like PCA or LDA) might be needed to make it effective.
  • System Calibration: The paper notes that system calibration, important for real-world deployment, was not covered.
  • Future Directions:
    • Explore feature-level contextualization alongside the DNN's internal contextualization.
  • Leverage side or meta-information during training to exploit complementary information in a condition-dependent manner.
  • Investigate various dimensionality reduction approaches for feature-level fusion.

Summary updated

BibTeX
@inproceedings{nandwana2018analysis,
  author = {Mahesh Kumar Nandwana and Mitchell McLaren and Diego Castan and Julien van Hout and Aaron Lawson},
  title = {{Analysis of Complementary Information Sources in the Speaker Embeddings Framework}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure