Abstract
Modern speaker verification systems often aggregate features from multiple network depths to capture both fine-grained acoustic details and abstract speaker characteristics. However, these intermediate features are typically not explicitly supervised. This paper introduces Multi-scale Feature Contrastive (MFCon) loss, which applies supervised contrastive learning to the intermediate layers of a Conformer-based model. By forcing the network to learn discriminative speaker features at every scale, the proposed method achieves a 2.41% Equal Error Rate (EER) on the VoxCeleb1-O benchmark, representing a 9.05% relative improvement over the baseline.
Introduction
Speaker verification aims to generate unique "speaker embeddings"—mathematical representations of a voice where embeddings from the same individual are clustered together and those from different individuals are separated. While Multi-scale Feature Aggregation (MFA) is a common technique to combine features from various depths, standard training only optimizes the final aggregated embedding. The authors argue that this approach misses the opportunity to refine intermediate representations, which could lead to a more robust final fingerprint.
Methods: Multi-scale Feature Contrastive (MFCon) Loss
The core contribution is the MFCon loss, integrated into an MFA-Conformer architecture consisting of six Conformer blocks.
- Intermediate Embedding Extraction: For each Conformer block, the feature map is processed through a dedicated pipeline: Layer Normalization, Attentive Statistics Pooling, and a Linear projection. This creates a distinct embedding for each scale.
- Supervised Contrastive (SupCon) Loss: A contrastive loss is applied to these intermediate embeddings, mathematically pulling same-speaker embeddings together and pushing different-speaker embeddings apart.
- Joint Optimization: The model is trained using a hybrid objective function that combines the traditional AM-Softmax classification loss on the final output with the MFCon loss on intermediate stages.
Experimental Setup
The model was trained on the VoxCeleb1 development set (1,211 speakers, 148k utterances) and evaluated using the Vox1-O protocol. Data augmentation included the MUSAN corpus for noise/music and the RIR database for reverberation simulation to ensure real-world robustness.
Results and Discussion
The researchers found that explicitly supervising intermediate layers prevents "feature drift" and ensures all parts of the network contribute effectively to speaker discrimination. Ablation studies indicated that using separate parameters for pooling and projection at each scale outperformed parameter sharing, suggesting that different scales require unique processing to extract optimal features.
The best performance was achieved by combining MFCon (intermediate supervision) with SupCon on the final embedding. This dual approach improves both the "raw materials" (intermediate features) and the "final product" (the speaker fingerprint).
As shown in the results, the Combined Loss configuration with optimized weights (lambda_1=0.03, lambda_2=0.03) yielded the lowest EER of 2.41% and a minDCF of 0.25. This significantly outperforms the MFA-Conformer baseline of 2.65% EER.
Conclusion
The MFCon framework demonstrates that multi-scale supervision significantly enhances speaker representation learning. While currently implemented on the Conformer architecture, the authors suggest this flexible approach could be extended to other models like ECAPA-TDNN. Future work may address the increased computational overhead during training to make this high-performance method more efficient.
Reference: Dixit, S., Baali, M., Singh, R., & Raj, B. (2024). Improving Speaker Representations Using Contrastive Losses on Multi-scale Features. arXiv:2410.05037v1 [cs.SD].
Summary updated