Differentiable Supervector Extraction for Encoding Speaker and Phrase Information in Text Dependent Speaker Verification

Victoria Mingote, Antonio Miguel, Alfonso Ortega, Eduardo Lleida

Introduction

Speaker verification systems are broadly categorized into text-independent and text-dependent tasks. In text-dependent verification, the user must recite a specific passphrase, adding a layer of security based on phonetic content. While modern deep learning has excelled in text-independent tasks using mean-pooling to create speaker embeddings, this approach is fundamentally flawed for text-dependent scenarios. Mean-pooling discards temporal information, making it impossible for the system to distinguish between different sequences of the same phonemes (e.g., "1-2-3" vs "3-2-1"). This paper addresses this limitation by introducing a differentiable alignment layer that preserves the temporal structure of speech.

Methodology: Differentiable Supervector Extraction

The core innovation of this work is the replacement of standard mean-pooling with a differentiable alignment mechanism. This allows the network to encode both speaker identity and phonetic sequence into a high-dimensional "supervector."

Figure 1: The proposed architecture integrates a 1D CNN front-end with an HMM-based alignment layer to extract temporal supervectors.

The process follows these key steps:

  1. Front-End Feature Extraction: Raw audio is converted into MFCCs and processed through a 1D Convolutional Neural Network (CNN). This extracts deep features while maintaining the temporal context of the frames.
  2. HMM Alignment: For each passphrase, a standard left-to-right Hidden Markov Model (HMM) is trained. Using the Viterbi algorithm, each acoustic frame is aligned to a specific phonetic state.
  3. Matrix-Based Alignment Layer: The alignment is represented as a binary matrix. The CNN output is multiplied by this matrix, effectively averaging frames per phonetic state rather than across the entire utterance.
  4. Supervector Formation: The state-specific averages are concatenated into a supervector. Because the matrix multiplication is differentiable, the entire pipeline—from the CNN to the supervector—can be trained end-to-end via backpropagation.

To improve robustness and prevent overfitting on small datasets, the authors also utilize Random Erasing as a data augmentation technique.

Experimental Setup

The proposed system was evaluated using the RSR2015 dataset (Part I), which includes recordings from 300 speakers across 30 different phrases. The background and development sets were used for training, while the evaluation set was reserved for enrollment and testing. Verification was performed using simple cosine similarity between supervectors.

Results and Discussion

The implementation of the differentiable alignment layer resulted in a dramatic performance boost. A baseline 3-layer 1D CNN using standard mean pooling achieved an Equal Error Rate (EER) of 11.70%. By simply swapping the pooling layer for the proposed alignment mechanism, the EER plummeted to 0.98%, representing a relative improvement of over 91%. With additional training data from the development set, the EER further improved to 0.72%.

Figure 6: t-SNE visualization of the proposed supervectors, demonstrating clear clustering by both speaker and phrase.

Visual analysis using t-SNE confirms these quantitative gains. While traditional mean embeddings fail to separate different phrases from the same speaker, the proposed supervectors form distinct, well-separated clusters for every unique phrase-speaker combination, as shown in Figure 6.

Conclusion

This research demonstrates that preserving temporal structure is essential for text-dependent speaker verification. By bridging traditional HMM alignment with modern deep learning through a differentiable matrix operation, the authors achieved a state-of-the-art reduction in error rates. Future work may involve exploring more complex backends like PLDA or more sophisticated alignment techniques to further refine the system's accuracy and robustness.

Summary updated

BibTeX
@inproceedings{mingote2018differentiable,
  author = {Victoria Mingote and Antonio Miguel and Alfonso Ortega and Eduardo Lleida},
  title = {{Differentiable Supervector Extraction for Encoding Speaker and Phrase Information in Text Dependent Speaker Verification}},
  booktitle = {IberSPEECH},
  year = {2018},
  doi = {10.21437/iberspeech.2018-1},
}
Figure