Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association

Wuyang Chen, Yanjie Sun, Kele Xu, Yong Dou

Introduction

Humans possess a remarkable innate ability to infer facial characteristics from vocal cues, a phenomenon known as voice-face association. In the field of artificial intelligence, replicating this cross-modal capability is a significant challenge, particularly in multilingual environments where linguistic variations in prosody—rhythm, stress, and intonation—can obscure underlying biometric traits. Furthermore, real-world "in-the-wild" data often contains significant noise, such as overlapping voices or poor lighting, which degrades model performance.

Researchers from the National University of Defense Technology recently presented a robust framework at ACM Multimedia 2024 to address these issues. Their approach, which earned 2nd place in the FAME 2024 Challenge, utilizes a combination of Supervised Cross-Contrastive (SCC) learning and a novel "chaining-cluster" post-processing technique to achieve language-agnostic and noise-robust association.

The overall framework of the proposed method, illustrating the two-phase approach: Supervised Cross-Contrastive Learning and Chaining-Cluster post-processing.

Methodology

The proposed system operates in two distinct phases: a training phase focused on feature alignment and a testing phase focused on score refinement.

Supervised Cross-Contrastive (SCC) Learning

The core architecture consists of a two-branch network featuring voice and face encoders, each utilizing 11 Transformer layers. To bridge the gap between modalities, the researchers introduced a shared-weight Transformer layer between the branches. This design encourages the encoders to map disparate sensory inputs into a unified embedding space.

During training, the model employs a contrastive loss function. By defining positive samples as the same identity across different languages (e.g., Hindi, English, and Urdu), the network is forced to ignore linguistic shifts and focus on invariant biometric features. This ensures that the resulting embeddings are language-invariant.

Chaining-Cluster Post-processing

To handle the inherent noise in real-world datasets, the researchers developed a hierarchical "chaining-cluster" strategy applied during inference. This process involves three key steps:

  1. Gender Clustering: Using K-Means, the system initially separates voice and face representations into male and female groups. Samples identified as outliers—those significantly distant from the cluster centroids—are filtered to improve reliability.
  2. Identity Clustering: Within the gender-specific groups, a finer clustering step groups samples by individual identity. This helps identify "prototypes" for each person.
  3. Score Refinement: The initial similarity scores generated by the SCC network are adjusted based on cluster consistency. If a predicted match involves a gender mismatch, the score is penalized. Conversely, matches between high-confidence identity clusters are rewarded.
An analysis of cluster distance metrics used to determine the optimal number of clusters for the post-processing phase.

Experimental Results

The framework was evaluated on the MAV-Celeb dataset, which features celebrities speaking English, Hindi, and Urdu. Performance was measured using the Equal Error Rate (EER), where lower values indicate higher accuracy.

  • FAME 2024 Challenge (V1): The system achieved an EER of 20.5, securing a global 2nd-place ranking.
  • MAV-Celeb V2: The model recorded an EER of 20.4, outperforming the Fusion and Orthogonal Projection (FOP) baseline of 22.0.
  • Ablation Study: The impact of the chaining-cluster technique was profound. Without the score refinement step, the EER on the V1 dataset increased from 20.5 to 29.6, highlighting the necessity of clustering for managing noisy, real-world data.

Conclusion

This research demonstrates that combining deep contrastive learning with structured post-processing can effectively bridge the gap between facial and vocal modalities across languages. By leveraging identity prototypes and gender consistency, the chaining-cluster method provides a significant boost in robustness. Future work will explore end-to-end architectures that can natively handle outliers and multilingual shifts without the need for multi-step post-processing.

Summary updated

BibTeX
@inproceedings{chen2024contrastive,
  author = {Wuyang Chen and Yanjie Sun and Kele Xu and Yong Dou},
  title = {{Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3688981},
}
Figure