Introduction
Speaker verification is a critical component of biometric security systems. While auditory speech has long been the primary modality, visual speech (lip movements) offers a robust complementary signal, particularly in noisy environments. Traditional multi-modal systems often rely on late fusion, where audio and video are processed independently and combined only at the decision stage. However, this approach fails to exploit the inherent physical synchronization and correlation between speech and lip movements.
This paper introduces a cross-modal co-learning framework designed to allow audio and visual modalities to mutually enhance each other during the training process. By addressing challenges such as temporal mismatch and feature noise, the proposed system achieves significant performance gains over both unimodal and standard fusion baselines.
Methodology: The Co-learning Framework
The core of the proposed system is a "pseudo-siamese" architecture that facilitates information exchange between modalities. While the encoders—ECAPA-TDNN for audio and a multi-stage CNN for visual—extract primary features, the Cross-modal Booster acts as the engine for mutual improvement.
The MaxFormer Block
The booster utilizes a stack of MaxFormer blocks, which address two primary hurdles:
- Temporal Alignment: A Multi-head Cross-Attention mechanism aligns sequences of different sampling rates by treating one modality as a "query" and the other as "keys/values."
- Noise Suppression: The Max-Feature-Map (MFM) operation performs local feature selection. By activating only the most competitive values, MFM filters out distortions that might arise during cross-modal transfer.
The model is optimized using a Co-learning Loss, which aggregates four Additive Angular Margin Softmax (AAMSoftmax) losses: two for the original unimodal features and two for the enhanced "transferred" features.
Experimental Setup and Datasets
The framework was evaluated across several diverse benchmarks:
- In-the-wild: LRSLip3 and VoxLip datasets, featuring real-world conditions with varying visual resolutions.
- Controlled Lab: GridLip and LomGridLip datasets, providing high-resolution visual data.
- Vox1-O-29: A custom subset of VoxCeleb1 specifically curated for this verification task.
Results and Discussion
The co-learning approach demonstrated a 60% average relative improvement over unimodal systems and a 20% improvement over standard late fusion methods. On the high-resolution GridLip-O dataset, the system achieved an Equal Error Rate (EER) of 0.45%.
Error Analysis and Mutual Improvement
A key finding was that "transferred" features often outperformed their original unimodal counterparts. This suggests the model successfully learns complementary identity characteristics.
As shown in the Venn diagrams, there is a distinct set of cases (Area I) where neither the original audio (A) nor visual (V) models could correctly verify the speaker, but the co-learning model (A') succeeded. This confirms that the cross-modal booster creates a unique, more discriminative feature space.
Conclusion
The proposed cross-modal co-learning framework effectively bridges the gap between auditory and visual speech for speaker verification. By allowing modalities to "teach" each other through the MaxFormer booster, the system achieves state-of-the-art robustness. Future research will explore extending this paradigm to text-dependent verification scenarios, further enhancing security for voice-password applications.
Summary updated