Abstract
The paper aims to address the task of speaker verification with single-channel, noisy and far-field speech by learning an embedding or feature representation that is invariant to different acoustic environments. We approach from two different directions. First, we adopt a newly proposed discriminative model that hybridizes Deep Neural Network (DNN) and Total Variability Model (TVM) with the goal of integrating their strengths. DNN helps learning a unique variable length representation of the feature sequence while TVM accumulates them into a fixed dimensional vector. Second, we propose a multitask training scheme with cross entropy and triplet losses in order to obtain good classification performance as well as distinctive speaker embeddings. The multi-task training is applied on both the DNN-TVM model and state-of-the-art x-vector system. The results on the development and evaluation sets of the VOiCES challenge reveal that the proposed multi-task training helps improving models that are solely based on cross entropy, and it works better with DNN-TVM architecture than x-vector for the current task. Moreover, the multi-task models tend to show complementary relationship with cross entropy models, and thus improved performance is observed after fusion.
Introduction
Speaker Verification (SV)—the process of confirming if a voice belongs to a specific person—is notoriously difficult in "in-the-wild" conditions. When a speaker is far from the microphone (far-field) or surrounded by background noise and reverberation, the acoustic signal becomes distorted.
Traditional systems like i-vectors (based on generative models) are robust but often lack the discriminative power of modern deep learning. Conversely, modern x-vectors (DNN-based) are powerful but typically rely on simple "global statistics pooling" (mean and standard deviation) to turn variable-length speech into a fixed-length signature. This research seeks to bridge these two worlds to create a speaker embedding that is invariant to harsh acoustic environments.
Methodology
The proposed approach introduces two primary innovations to enhance speaker embeddings:
1. Hybrid DNN-TVM Architecture
Instead of the standard x-vector approach, the authors use a Time Delay Neural Network (TDNN) to extract frame-level features. They then replace the standard pooling layer with a Total Variability Model (TVM) layer. This layer calculates "Baum-Welch statistics"—a more sophisticated way of aggregating features that was traditionally used in i-vector systems. Crucially, they formulate this as a "distribution-free" model, meaning it doesn't rely on the rigid Gaussian assumptions of older systems, allowing the entire network to be trained end-to-end.
2. Multi-task Discriminative Training
Most SV systems are trained using Cross-Entropy (CE) loss, which focuses on classifying the speaker correctly. However, CE doesn't explicitly force the "embeddings" (the digital voice fingerprints) of the same speaker to be close together in mathematical space. The authors add a Triplet Loss to the training process. This loss specifically looks at triplets of samples (an anchor, a positive match, and a negative mismatch) and pushes the anchor closer to the positive while shoving the negative further away.
Experimental Setup
- Training Data: A massive "in-the-wild" collection including VoxCeleb 1 & 2 and Speakers in the Wild (SITW).
- Data Augmentation: To simulate the VOiCES challenge conditions, the training data was augmented with television noise, babble, music, and 60,000 different simulated Room Impulse Responses (RIRs) to mimic reverberation.
- Evaluation: The VOiCES from a Distance Challenge 2019 corpus, specifically designed to test far-field speech recorded at various distances and angles in noisy rooms.
Key Results
The primary metric used was minDCF (Minimum Detection Cost Function), where lower values indicate better performance. The experimental results demonstrate the efficacy of both the hybrid architecture and the multi-task training strategy.
- Multi-task Improvement: On the VOiCES evaluation set, multi-task training improved the DNN-TVM model's minDCF by 28% (from 0.89 to 0.64) compared to the version trained only with cross-entropy.
- Hybrid vs. Standard: The DNN-TVM multi-task model outperformed the native x-vector multi-task implementation by approximately 6% in terms of minDCF (0.64 vs 0.68).
- System Fusion: The best-performing setup was a fusion of the Kaldi x-vector baseline and the multi-task models, achieving a 3% improvement over the highly optimized state-of-the-art Kaldi baseline on the evaluation set (0.60 vs 0.62).
Discussion and Insights
- Pooling as a Bottleneck: The transition from simple global mean pooling to a TVM-based layer suggests that the method of aggregating temporal information is as critical as the feature extraction itself.
- Complementary Loss Functions: Cross-Entropy and Triplet Loss address different aspects of the problem. CE provides a global structure for classification, while Triplet Loss fine-tunes the embedding space to reduce intra-speaker variance.
- The Power of Fusion: The hybrid DNN-TVM model provided complementary information to the standard x-vector. Fusing scores from both systems yielded superior performance, suggesting they capture distinct, useful characteristics of the voice data.
Limitations and Future Work
Despite the improvements, several challenges remain. The authors noted that resource constraints limited training to a subset of the augmented dataset (~2.1 million utterances out of 6.5 million). Additionally, a performance gap exists between the Keras-based x-vector and the industry-standard Kaldi implementation, likely due to low-level optimizations in Kaldi. Future work will focus on scaling to the full dataset and investigating the implementation gap to further refine the hybrid architecture.
Summary updated