Self-Supervised Speaker Embeddings

Themos Stafylakis, Johan Rohdin, Oldřich Plchot, Petr Mizera, Lukáš BurgetInterspeech 2019

TL;DR

This paper introduces a novel self-supervised learning approach to train speaker embeddings without relying on labeled speaker data. By training a decoder to reconstruct speech segments using the speaker embedding and estimated phone sequences, they achieve competitive or improved performance compared to traditional supervised methods, especially in semi-supervised scenarios.

What Problem Does This Paper Address?

Traditional speaker recognition systems, particularly those using deep learning like x-vectors, heavily rely on large datasets with labeled speaker identities for training. This poses a significant challenge, as obtaining and annotating such datasets is costly and time-consuming. While older methods like i-vectors could leverage unlabeled data, they were less performant. This paper aims to bridge this gap by developing a method to train powerful speaker embeddings using primarily unlabeled speech data, making speaker recognition more accessible and scalable.

Methods

The core innovation lies in augmenting a standard speaker embedding extractor (like an x-vector network) with a decoder. This decoder is trained to reconstruct a speech segment from the same utterance, conditioned on two key pieces of information:

  • Speaker Embedding: The output of the standard speaker embedding extractor.
  • Estimated Phone Sequence: The sequence of phonetic units for the target speech segment, predicted by a separate, pre-trained Automatic Speech Recognition (ASR) model.

This "reconstruction loss" can be used as a standalone training objective (purely self-supervised) or combined with the traditional cross-entropy loss over labeled speakers. The latter acts as a regularizer, improving generalization. The authors also explore using different segments for encoding and decoding, and conditioning the decoder on phonetic context (e.g., a 7-frame window) rather than just the current frame's phone.

Datasets

  • VoxCeleb 1: Used for training and evaluation.
  • Speakers in the Wild (SITW): Used for evaluation.
  • SITW core-core development set: Used for hyperparameter tuning.

The paper notes that for training, they used a large dataset derived from VoxCeleb 1 and 2, containing 7146 speakers and over 2 million utterances (including augmentations). For semi-supervised experiments, they randomly selected 1000 speakers from this set.

Key Insights and Ideas

  • Self-supervision for speaker identity: The paper demonstrates that reconstructing speech segments can effectively teach a network to capture speaker-discriminative information, even without explicit speaker labels.
  • Phonetic conditioning is crucial: Conditioning the decoder on the phone sequence, in addition to the speaker embedding, is vital. This allows the embedding to focus on how a speaker pronounces sounds, rather than what sounds are being pronounced, leading to better speaker discrimination.
  • Denoising effect: Using clean decoding targets while encoding potentially augmented segments encourages the embedding extractor to learn robust, denoising features.
  • Regularization power: Combining the self-supervised reconstruction loss with the standard cross-entropy loss acts as a powerful regularizer, improving generalization to unseen speakers.
  • Context matters: Using phonetic context (e.g., a 7-frame window) for the decoder input further enhances performance.

Key Results

  • Improved performance in semi-supervised settings: When only a small fraction of training data is labeled, the proposed self-supervised approach significantly outperforms standard x-vectors trained with limited labels.
  • Competitive performance with fully supervised methods: Even when trained purely in a self-supervised manner, the proposed method achieves comparable or slightly better results than fully supervised x-vectors on some benchmarks.
  • Combined loss yields best results: Combining the self-supervised loss with the cross-entropy loss (Spk+Self) consistently yields the best performance across various configurations, particularly in terms of Minimum Detection Cost (mDCF).
  • Contextual conditioning benefits: Using a 7-frame phonetic context (ctx) generally leads to better results than frame-level conditioning alone.

Limitations and Future Work

  • Reliance on ASR: The method requires a pre-trained ASR model to provide phone sequences. While this model is only needed during training, its accuracy can impact the effectiveness of the self-supervised training.
  • Future Directions:
    • Revisiting other self-supervised methods like variational autoencoders in an end-to-end fashion.
  • Jointly training embedding extractors with Text-to-Speech (TTS) systems from scratch.
  • Exploring more advanced decoder architectures (recurrent, attention-based) and encoder depths.
  • Investigating different segment durations for encoding and decoding.
  • Extending the approach to leverage speaker labels more effectively in weakly supervised settings (e.g., when segments from different utterances of the same speaker are known).

Summary updated

BibTeX
@inproceedings{stafylakis2019self,
  author = {Themos Stafylakis and Johan Rohdin and Oldřich Plchot and Petr Mizera and Lukáš Burget},
  title = {{Self-Supervised Speaker Embeddings}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure