Bidirectional Attention for Text-Dependent Speaker Verification

Xin Fang, Tian Gao, Liang Zou, Zhen-Hua Ling

TL;DR

This paper introduces a novel "bidirectional attention" mechanism for text-dependent speaker verification (TDSV). By jointly considering enrollment and evaluation utterances, their approach creates dynamic, interactive speaker representations that significantly outperform existing methods, achieving a competitive 6.26% Equal Error Rate (EER).

What Problem Does This Paper Address?

Speaker verification, especially text-dependent (TDSV), aims to confirm a person's identity based on their voice using a specific phrase. While deep learning has advanced TDSV, current methods often treat the "enrollment" (reference) speaker's voice features as static. This overlooks how a human listener might subtly adjust their focus on different aspects of the reference voice when comparing it to various test utterances. The paper addresses this by proposing a method that dynamically models the interaction between enrollment and evaluation utterances, creating more robust and context-aware speaker representations.

Methods

The core innovation is a bidirectional attention mechanism integrated into a Convolutional Neural Network (CNN) framework. Here's a breakdown:

  • CNN-based Feature Extraction (NET1): A CNN is used to extract frame-level acoustic features from both enrollment and evaluation utterances.
  • Utterance-Level Feature Aggregation (NET2): These frame-level features are then aggregated into utterance-level representations. These are considered "constant" features as they are derived independently for each utterance.
  • Bidirectional Attention (NET3): This is the key novelty. Instead of treating enrollment features as fixed, NET3 uses the frame-level features of one utterance (e.g., enrollment) and the utterance-level features of the other (e.g., evaluation) to compute attention weights. This allows the model to learn how to "attend" to specific parts of the enrollment utterance based on the context of the evaluation utterance, creating interactive speaker representations (EnR and EvR).
  • Metric Learning (NET4): A final layer combines the interactive representations (EnR, EvR) with the constant utterance-level features (EnH, EvH) to make the final verification decision.
  • Phonetic Loss Function: An additional loss function is introduced to specifically leverage phonetic information, aiming to improve the accuracy of attention score calculation and phonetic-specific pooling.
  • Circle Loss: The paper replaces the traditional triplet loss with a "circle loss" for NET2, which offers more flexible optimization by dynamically adjusting penalty strengths for within-class and between-class similarities.

Datasets

The proposed method was evaluated on an internal benchmark dataset called "Dan Dan Ni Hao". This dataset consists of audio recordings collected from three homemade sessions at iFlytek. The paper details its use for training, development (validation), and testing, with specific numbers of speakers and trials provided for each set. No new benchmark dataset was introduced.

Key Insights and Ideas

  • The "fixed reference" is a limitation: The paper convincingly argues that treating enrollment speaker features as static is a bottleneck in TDSV. Human listeners adapt their focus, and this model mimics that by creating dynamic, interactive representations.
  • Joint information is crucial: The bidirectional attention mechanism effectively captures the complementary information between enrollment and evaluation utterances, leading to better performance than using either independently.
  • Phonetics matter for attention: Incorporating phonetic information through a dedicated loss function helps the attention mechanism focus more precisely on relevant speech segments.
  • Circle loss offers better optimization: Replacing triplet loss with circle loss provides a more flexible and potentially more stable optimization process for learning speaker-discriminative features.

Key Results

  • The proposed Bidirectional Attention CNN (BaCNN) model achieved a competitive Equal Error Rate (EER) of 6.26% on the internal "Dan Dan Ni Hao" benchmark dataset.
  • This performance outperformed various baseline methods, including traditional i-vector/PLDA, d-vector, self-attention, and sequence-to-sequence attention models.
  • Specifically, the BaCNN model showed a relative decrease in EER of 15.61% compared to the d-vector baseline system on the development set.
  • The introduction of phonetic information (using the softmax cross-entropy loss of NET1) led to a relative decrease in EER of 5.0% and 5.44% on the development and test sets, respectively, compared to models neglecting it.

Limitations and Future Work

  • Dataset Specificity: The primary evaluation was on an internal dataset. While promising, performance on more diverse, large-scale datasets would be valuable.
  • Computational Cost: While not explicitly detailed as a limitation, deep learning models with attention mechanisms can be computationally intensive.
  • Future Directions: The authors suggest that the proposed BaCNN approach, which mimics human selective auditory attention, could also be applied to text-independent speaker verification tasks.

Summary updated

Abstract

Automatic speaker verification provides a flexible and effective way for biometric authentication. Previous deep learning-based methods have demonstrated promising results, whereas a few problems still require better solutions. In prior works examining speaker discriminative neural networks, the speaker representation of the target speaker is regarded as a fixed one when comparing with utterances from different speakers, and the joint information between enrollment and evaluation utterances is ignored. In this paper, we propose to combine CNN-based feature learning with a bidirectional attention mechanism to achieve better performance with only one enrollment utterance. The evaluation-enrollment joint information is exploited to provide interactive features through bidirectional attention. In addition, we introduce one individual cost function to identify the phonetic contents, which contributes to calculating the attention score more specifically. These interactive features are complementary to the constant ones, which are extracted from individual speakers separately and do not vary with the evaluation utterances. The proposed method archived a competitive equal error rate of 6.26% on the internal “DAN DAN NI HAO” benchmark dataset with 1250 utterances and outperformed various baseline methods, including the traditional i-vector/PLDA, d-vector, self-attention, and sequence-to-sequence attention models.
BibTeX
@inproceedings{fang2020bidirectional,
  author = {Xin Fang and Tian Gao and Liang Zou and Zhen-Hua Ling},
  title = {{Bidirectional Attention for Text-Dependent Speaker Verification}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=E25MNNRY3S},
}
Figure