Improving Speaker Representations Using Contrastive Losses on Multi-scale Features

Satvik Dixit, Massa Baali, Rita Singh, Bhiksha RajarXiv 2024 · DOI · arXiv

Abstract

Modern speaker verification systems often aggregate features from multiple network depths to capture both fine-grained acoustic details and abstract speaker characteristics. However, these intermediate features are typically not explicitly supervised. This paper introduces Multi-scale Feature Contrastive (MFCon) loss, which applies supervised contrastive learning to the intermediate layers of a Conformer-based model. By forcing the network to learn discriminative speaker features at every scale, the proposed method achieves a 2.41% Equal Error Rate (EER) on the VoxCeleb1-O benchmark, representing a 9.05% relative improvement over the baseline.

Introduction

Speaker verification aims to generate unique "speaker embeddings"—mathematical representations of a voice where embeddings from the same individual are clustered together and those from different individuals are separated. While Multi-scale Feature Aggregation (MFA) is a common technique to combine features from various depths, standard training only optimizes the final aggregated embedding. The authors argue that this approach misses the opportunity to refine intermediate representations, which could lead to a more robust final fingerprint.

Methods: Multi-scale Feature Contrastive (MFCon) Loss

The core contribution is the MFCon loss, integrated into an MFA-Conformer architecture consisting of six Conformer blocks.

  1. Intermediate Embedding Extraction: For each Conformer block, the feature map is processed through a dedicated pipeline: Layer Normalization, Attentive Statistics Pooling, and a Linear projection. This creates a distinct embedding for each scale.
  2. Supervised Contrastive (SupCon) Loss: A contrastive loss is applied to these intermediate embeddings, mathematically pulling same-speaker embeddings together and pushing different-speaker embeddings apart.
  3. Joint Optimization: The model is trained using a hybrid objective function that combines the traditional AM-Softmax classification loss on the final output with the MFCon loss on intermediate stages.
Architecture of the MFA-Conformer with MFCon loss applied to intermediate layers.

Experimental Setup

The model was trained on the VoxCeleb1 development set (1,211 speakers, 148k utterances) and evaluated using the Vox1-O protocol. Data augmentation included the MUSAN corpus for noise/music and the RIR database for reverberation simulation to ensure real-world robustness.

Results and Discussion

The researchers found that explicitly supervising intermediate layers prevents "feature drift" and ensures all parts of the network contribute effectively to speaker discrimination. Ablation studies indicated that using separate parameters for pooling and projection at each scale outperformed parameter sharing, suggesting that different scales require unique processing to extract optimal features.

The best performance was achieved by combining MFCon (intermediate supervision) with SupCon on the final embedding. This dual approach improves both the "raw materials" (intermediate features) and the "final product" (the speaker fingerprint).

Performance comparison of different loss configurations on the VoxCeleb1-O benchmark.

As shown in the results, the Combined Loss configuration with optimized weights (lambda_1=0.03, lambda_2=0.03) yielded the lowest EER of 2.41% and a minDCF of 0.25. This significantly outperforms the MFA-Conformer baseline of 2.65% EER.

Conclusion

The MFCon framework demonstrates that multi-scale supervision significantly enhances speaker representation learning. While currently implemented on the Conformer architecture, the authors suggest this flexible approach could be extended to other models like ECAPA-TDNN. Future work may address the increased computational overhead during training to make this high-performance method more efficient.

Reference: Dixit, S., Baali, M., Singh, R., & Raj, B. (2024). Improving Speaker Representations Using Contrastive Losses on Multi-scale Features. arXiv:2410.05037v1 [cs.SD].

Summary updated

Abstract

Speaker verification systems have seen significant advancements with the introduction of Multi-scale Feature Aggregation (MFA) architectures, such as MFA-Conformer and ECAPA-TDNN. These models leverage information from various network depths by concatenating intermediate feature maps before the pooling and projection layers, demonstrating that even shallower feature maps encode valuable speaker-specific information. Building upon this foundation, we propose a Multi-scale Feature Contrastive (MFCon) loss that directly enhances the quality of these intermediate representations. Our MFCon loss applies contrastive learning to all feature maps within the network, encouraging the model to learn more discriminative representations at the intermediate stage itself. By enforcing better feature map learning, we show that the resulting speaker embeddings exhibit increased discriminative power. Our method achieves a 9.05% improvement in equal error rate (EER) compared to the standard MFA-Conformer on the VoxCeleb-1O test set.
BibTeX
@misc{dixit2024improving,
  author = {Satvik Dixit and Massa Baali and Rita Singh and Bhiksha Raj},
  title = {{Improving Speaker Representations Using Contrastive Losses on Multi-scale Features}},
  year = {2024},
  eprint = {2410.05037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05037},
}
Figure