Towards a Fault-Tolerant Speaker Verification System: A Regularization Approach to Reduce the Condition Number

Siqi Zheng, Gang Liu, Hongbin Suo, Yun LeiInterspeech 2019 · DOI

Abstract

In real-world smart speaker deployments, training data is often harvested automatically, leading to significant label noise when devices are shared among multiple users. This paper introduces a fault-tolerant speaker verification framework designed to mitigate the impact of mislabeled data. By focusing on reducing the model's "condition number" through entropy loss regularization, segment reshuffling, and co-training, the authors achieve a relative error rate reduction of over 35% on a large-scale corpus of 22 million utterances.

Introduction

The rapid proliferation of smart speakers has enabled the collection of massive speech datasets. However, unlike human-curated datasets like NIST SRE, these "in-the-wild" datasets often rely on device IDs for labeling. Because multiple individuals frequently share a single device, the resulting training data is often mislabeled. Previous research indicates that even a 10% mislabeling rate can degrade the Equal Error Rate (EER) of x-vector systems from ~2.7% to nearly 13%. This paper proposes a robust training methodology to maintain high verification accuracy despite such label noise.

Methodology

The authors propose improving "fault tolerance" by minimizing the system's Condition Number—a metric representing the sensitivity of the model's output to small input perturbations, such as incorrect labels. Three key strategies are employed:

Entropy Loss Regularization

Standard cross-entropy loss assumes "hard" ground truth labels. To handle uncertainty, the authors introduce a regularized loss function that "softens" the penalty for mislabeled samples. This prevents the model from making drastic, potentially incorrect weight updates during backpropagation when encountering noisy data.

Regularization by Segments Reshuffling

A novel "lemonade from lemons" approach is used for low-confidence utterances. Instead of discarding them, these utterances are partitioned into segments and appended to high-confidence utterances from the same speaker. This process effectively treats the mislabeled voice as realistic background noise, helping the model learn to distinguish the target speaker from common interference.

The process of segment reshuffling to augment high-confidence utterances with segments from less confident ones.

Co-Training

The system utilizes two separate classifiers that view the data through distinct feature spaces. By sharing high-confidence labels between the two models, each classifier acts as a regularizer for the other, enhancing the overall robustness of the verification system.

Experimental Results

The proposed framework was evaluated on the NIST SRE (04-10) dataset with synthetic noise and the Tmall Genie Corpus, which contains 22 million real-world utterances.

Comparison of EER between baseline and regularized models under different mislabeling conditions.

As shown in the results, the regularization techniques provide substantial gains. On the NIST SRE dataset with 10% mislabeled data, the EER dropped from 12.81% to 6.95%. On the massive Tmall Genie corpus, the proposed system reduced the EER from a baseline of 9.80% to 6.33%, representing a relative improvement of over 35%. These results confirm that combining model-level regularization with data-level reshuffling is highly effective for large-scale, noisy datasets.

Conclusion

This work demonstrates that mathematical regularization of the condition number is a viable alternative to exhaustive data cleaning in large-scale speaker verification. By treating mislabeled data as a resource for augmentation rather than a liability, the proposed system achieves high fault tolerance. Future research will explore more sophisticated labeling quality measures to further refine the training process.

Summary updated

BibTeX
@inproceedings{zheng2019towards,
  author = {Siqi Zheng and Gang Liu and Hongbin Suo and Yun Lei},
  title = {{Towards a Fault-Tolerant Speaker Verification System: A Regularization Approach to Reduce the Condition Number}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1442},
}
Figure