Introduction
Humans possess a remarkable innate ability to infer facial characteristics from vocal cues, a phenomenon known as voice-face association. In the field of artificial intelligence, replicating this cross-modal capability is a significant challenge, particularly in multilingual environments where linguistic variations in prosody—rhythm, stress, and intonation—can obscure underlying biometric traits. Furthermore, real-world "in-the-wild" data often contains significant noise, such as overlapping voices or poor lighting, which degrades model performance.
Researchers from the National University of Defense Technology recently presented a robust framework at ACM Multimedia 2024 to address these issues. Their approach, which earned 2nd place in the FAME 2024 Challenge, utilizes a combination of Supervised Cross-Contrastive (SCC) learning and a novel "chaining-cluster" post-processing technique to achieve language-agnostic and noise-robust association.
Methodology
The proposed system operates in two distinct phases: a training phase focused on feature alignment and a testing phase focused on score refinement.
Supervised Cross-Contrastive (SCC) Learning
The core architecture consists of a two-branch network featuring voice and face encoders, each utilizing 11 Transformer layers. To bridge the gap between modalities, the researchers introduced a shared-weight Transformer layer between the branches. This design encourages the encoders to map disparate sensory inputs into a unified embedding space.
During training, the model employs a contrastive loss function. By defining positive samples as the same identity across different languages (e.g., Hindi, English, and Urdu), the network is forced to ignore linguistic shifts and focus on invariant biometric features. This ensures that the resulting embeddings are language-invariant.
Chaining-Cluster Post-processing
To handle the inherent noise in real-world datasets, the researchers developed a hierarchical "chaining-cluster" strategy applied during inference. This process involves three key steps:
- Gender Clustering: Using K-Means, the system initially separates voice and face representations into male and female groups. Samples identified as outliers—those significantly distant from the cluster centroids—are filtered to improve reliability.
- Identity Clustering: Within the gender-specific groups, a finer clustering step groups samples by individual identity. This helps identify "prototypes" for each person.
- Score Refinement: The initial similarity scores generated by the SCC network are adjusted based on cluster consistency. If a predicted match involves a gender mismatch, the score is penalized. Conversely, matches between high-confidence identity clusters are rewarded.
Experimental Results
The framework was evaluated on the MAV-Celeb dataset, which features celebrities speaking English, Hindi, and Urdu. Performance was measured using the Equal Error Rate (EER), where lower values indicate higher accuracy.
- FAME 2024 Challenge (V1): The system achieved an EER of 20.5, securing a global 2nd-place ranking.
- MAV-Celeb V2: The model recorded an EER of 20.4, outperforming the Fusion and Orthogonal Projection (FOP) baseline of 22.0.
- Ablation Study: The impact of the chaining-cluster technique was profound. Without the score refinement step, the EER on the V1 dataset increased from 20.5 to 29.6, highlighting the necessity of clustering for managing noisy, real-world data.
Conclusion
This research demonstrates that combining deep contrastive learning with structured post-processing can effectively bridge the gap between facial and vocal modalities across languages. By leveraging identity prototypes and gender consistency, the chaining-cluster method provides a significant boost in robustness. Future work will explore end-to-end architectures that can natively handle outliers and multilingual shifts without the need for multi-step post-processing.
Summary updated