Abstract
In this paper, we propose an iterative framework for self-supervised speaker representation learning based on a deep neural network (DNN). The framework starts with training a self-supervision speaker embedding network by maximizing agreement between different segments within an utterance via a contrastive loss. Taking advantage of DNN's ability to learn from data with label noise, we propose to cluster the speaker embedding obtained from the previous speaker network and use the subsequent class assignments as pseudo labels to train a new DNN. Moreover, we iteratively train the speaker network with pseudo labels generated from the previous step to bootstrap the discriminative power of a DNN. Speaker verification experiments are conducted on the VoxCeleb dataset. The results show that our proposed iterative self-supervised learning framework outperformed previous works using self-supervision. The speaker network after 5 iterations obtains a 61% performance gain over the speaker embedding model trained with contrastive loss.
Introduction
Deep learning has revolutionized speaker recognition, but these models typically require massive datasets where every clip is manually labeled with a speaker ID. Labeling thousands of hours of audio is expensive, slow, and prone to human error. While self-supervised methods—where models learn from unlabeled data—offer a solution, they often hit a performance ceiling. A common technique is contrastive learning, which assumes segments from the same recording belong to the same person. However, in real-world data, the same person often appears across different recordings, creating "label noise" that limits model accuracy.
The Iterative Framework
To overcome these limitations, we propose a three-stage iterative framework designed to bootstrap discriminative power from unlabeled data:
- Initial Round (Contrastive Learning): The model learns basic features using a contrastive loss (CSL), ensuring snippets from the same utterance have similar mathematical signatures (embeddings).
- Clustering & Pseudo-Labeling: The model groups similar voices using k-means clustering. These cluster assignments serve as "pseudo-labels" for the next training phase.
- Label Purifying: To handle clustering errors, we filter out data points where the model is unconfident (based on distance to cluster centers) and discard clusters below a minimum size $S$.
- Iterative Training: The model is retrained as a classifier using these purified pseudo-labels. This cycle repeats; as the model improves, the clusters become more accurate, leading to higher-quality labels.
As shown in the table above, the Normalized Mutual Information (NMI) increases significantly after each purification step (indicated by the arrows), proving that the "self-evolution" strategy effectively reduces label noise.
Experimental Results
We trained our model on the VoxCeleb 2 development set (over 1 million utterances) without using ground-truth labels. Evaluation was performed on the VoxCeleb 1 test sets.
Our iterative approach showed dramatic improvements over the baseline. After 5 iterations, the model achieved a 61% performance gain over the initial contrastive learning model. On the VoxCeleb 1 test set, the Equal Error Rate (EER) dropped from 8.86% in the initial round to 3.45% by round 5.
As illustrated in the results table, our framework significantly outperforms other prominent self-supervised methods. While a fully supervised model achieves an EER of 1.51%, our self-supervised method is narrowing the gap considerably, demonstrating the robustness of DNNs when trained on "mostly right" pseudo-labels.
Conclusion and Future Work
This work demonstrates that an iterative cycle of clustering and purification can effectively bootstrap speaker representations from unlabeled data. Future research will focus on improving training efficiency through continual learning—updating the model incrementally rather than from scratch—and exploring automated clustering methods to remove the need for pre-defining the number of speakers.
Summary updated