Phoneme-aware and Channel-wise Attentive Learning for Text DependentSpeaker Verification

Yan Liu, Zheng Li, Lin Li, Qingyang Hong

Abstract

Text-dependent speaker verification (SV) requires distinguishing speaker-specific vocal characteristics from fixed phonetic content. This paper proposes a multi-task learning (MTL) framework that leverages phonetic information as an attention mechanism to enhance speaker discrimination. By integrating phoneme-aware attentive pooling, Squeeze-and-Excitation (SE) blocks, and adversarial learning via a Gradient Reversal Layer (GRL), the proposed architecture effectively disentangles speaker identity from spoken content. Experimental results on the RSR2015 dataset demonstrate state-of-the-art performance, particularly in challenging impostor scenarios.

Introduction

Speaker Verification (SV) is the process of validating a claimed identity based on voice biometrics. In text-dependent SV, users are required to speak a specific passphrase. While the lexical constraint simplifies the task compared to text-independent SV, it introduces a significant challenge: the model must separate the content of the speech from the characteristics of the speaker.

Traditional "x-vector" systems often treat phonetic information as noise to be ignored in favor of a universal speaker representation. However, in text-dependent contexts, the idiosyncratic way an individual pronounces specific phonemes is a highly discriminative biometric feature. This research aims to utilize phonetic information not as a nuisance, but as a guide to focus the model on the most informative segments of an utterance.

Methodology

The proposed architecture is built upon the Extended Time-Delay Neural Network (E-TDNN) and incorporates several key innovations to improve feature extraction and disentanglement.

Multi-Task Learning and Architecture

The network is trained using a multi-task learning framework to simultaneously identify the speaker and the underlying phonemes. This approach ensures that the initial shared layers capture features relevant to both identity and linguistics.

Proposed multi-task learning architecture (S6) featuring SE-blocks, phoneme-aware pooling, and adversarial branches.

Phoneme-aware Attentive Pooling

Unlike standard statistics pooling that treats all frames equally, the authors introduce a phoneme-aware attentive pooling mechanism. Here, the phoneme posterior—the probability of a specific sound occurring at a given time—serves as an attention weight. This allows the model to selectively "attend" to phonetic segments that are most useful for speaker identification.

Detail of the Phoneme-aware Attentive Pooling layer, where phoneme posteriors weight the E-TDNN features.

Channel-wise Recalibration and Adversarial Learning

To further refine the features, Squeeze-and-Excitation (SE) blocks are integrated to perform channel-wise feature recalibration, emphasizing informative channels while suppressing noise. Additionally, to prevent the speaker embedding from simply memorizing the passphrase, a segment-level adversarial branch with a Gradient Reversal Layer (GRL) is employed. This branch attempts to predict phonemes at the segment level; during backpropagation, the gradient is flipped to "scrub" phonetic leakage from the speaker-specific layers.

Experimental Setup

The system was evaluated using the RSR2015 Part I corpus, which consists of 300 speakers (143 female, 157 male) providing 30 fixed phrase utterances. The dataset is partitioned into background (training), development, and evaluation sets. Trials are categorized into four types based on the correctness of the speaker and the content: Target-Correct (TC), Target-Wrong (TW), Impostor-Correct (IC), and Impostor-Wrong (IW).

Results and Discussion

The proposed system (S6) was benchmarked against standard i-vector and x-vector baselines. Performance was measured using the Equal Error Rate (EER).

On the RSR2015 Evaluation set (Male), the S6 system achieved an EER of 0.237% for Target-Wrong (TW) trials, a substantial improvement over the i-vector (1.950%) and standard x-vector (3.017%) approaches. Ablation studies confirmed that both SE-blocks and phoneme-aware pooling contributed to consistent error reductions across all trial categories. Notably, the system showed high robustness in "Impostor-Correct" (IC) trials—where an impostor speaks the correct passphrase—demonstrating superior speaker discrimination capabilities.

Conclusion

This work demonstrates that phonetic information can be effectively harnessed as an attention map to improve text-dependent speaker verification. By combining temporal attention (phoneme-aware pooling) with channel-wise attention (SE-blocks) and adversarial disentanglement, the model achieves high precision in separating speaker identity from linguistic content. Future work will explore the scalability of this approach to lexically-constrained scenarios where passphrases are not fixed.

Summary updated

BibTeX
@inproceedings{liu2021phoneme,
  author = {Yan Liu and Zheng Li and Lin Li and Qingyang Hong},
  title = {{Phoneme-aware and Channel-wise Attentive Learning for Text DependentSpeaker Verification}},
  booktitle = {Interspeech},
  year = {2021},
  doi = {10.21437/interspeech.2021-2137},
}
Figure