Cross-Modal Audio-Visual Co-Learning for Text-Independent Speaker Verification

Meng Liu, Kong Aik Lee, Longbiao Wang, Hanyi Zhang, Chang Zeng, Jianwu Dang

Introduction

Speaker verification is a critical component of biometric security systems. While auditory speech has long been the primary modality, visual speech (lip movements) offers a robust complementary signal, particularly in noisy environments. Traditional multi-modal systems often rely on late fusion, where audio and video are processed independently and combined only at the decision stage. However, this approach fails to exploit the inherent physical synchronization and correlation between speech and lip movements.

This paper introduces a cross-modal co-learning framework designed to allow audio and visual modalities to mutually enhance each other during the training process. By addressing challenges such as temporal mismatch and feature noise, the proposed system achieves significant performance gains over both unimodal and standard fusion baselines.

Methodology: The Co-learning Framework

The core of the proposed system is a "pseudo-siamese" architecture that facilitates information exchange between modalities. While the encoders—ECAPA-TDNN for audio and a multi-stage CNN for visual—extract primary features, the Cross-modal Booster acts as the engine for mutual improvement.

Architecture comparison: (a) Independent training baseline vs. (b) Proposed cross-modal co-learning with the MaxFormer block.

The MaxFormer Block

The booster utilizes a stack of MaxFormer blocks, which address two primary hurdles:

  1. Temporal Alignment: A Multi-head Cross-Attention mechanism aligns sequences of different sampling rates by treating one modality as a "query" and the other as "keys/values."
  2. Noise Suppression: The Max-Feature-Map (MFM) operation performs local feature selection. By activating only the most competitive values, MFM filters out distortions that might arise during cross-modal transfer.

The model is optimized using a Co-learning Loss, which aggregates four Additive Angular Margin Softmax (AAMSoftmax) losses: two for the original unimodal features and two for the enhanced "transferred" features.

Experimental Setup and Datasets

The framework was evaluated across several diverse benchmarks:

  • In-the-wild: LRSLip3 and VoxLip datasets, featuring real-world conditions with varying visual resolutions.
  • Controlled Lab: GridLip and LomGridLip datasets, providing high-resolution visual data.
  • Vox1-O-29: A custom subset of VoxCeleb1 specifically curated for this verification task.

Results and Discussion

The co-learning approach demonstrated a 60% average relative improvement over unimodal systems and a 20% improvement over standard late fusion methods. On the high-resolution GridLip-O dataset, the system achieved an Equal Error Rate (EER) of 0.45%.

Error Analysis and Mutual Improvement

A key finding was that "transferred" features often outperformed their original unimodal counterparts. This suggests the model successfully learns complementary identity characteristics.

Error analysis via Venn diagrams for LRSLip3 and GridLip. Area I represents cases where co-learning succeeded despite unimodal failures.

As shown in the Venn diagrams, there is a distinct set of cases (Area I) where neither the original audio (A) nor visual (V) models could correctly verify the speaker, but the co-learning model (A') succeeded. This confirms that the cross-modal booster creates a unique, more discriminative feature space.

Conclusion

The proposed cross-modal co-learning framework effectively bridges the gap between auditory and visual speech for speaker verification. By allowing modalities to "teach" each other through the MaxFormer booster, the system achieves state-of-the-art robustness. Future research will explore extending this paradigm to text-dependent verification scenarios, further enhancing security for voice-password applications.

Summary updated

Abstract

Visual speech (i.e., lip motion) is highly related to auditory speech due to the co-occurrence and synchronization in speech production. This paper investigates this correlation and proposes a cross-modal speech co-learning paradigm. The primary motivation of our cross-modal co-learning method is modeling one modality aided by exploiting knowledge from another modality. Specifically, two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Inside each booster, a max-feature-map embedded Transformer variant is proposed for modality alignment and enhanced feature generation. The network is co-learned both from scratch and with pretrained models. Experimental results on the test scenarios demonstrate that our proposed method achieves around 60% and 20% average relative performance improvement over baseline unimodal and fusion systems, respectively.
BibTeX
@inproceedings{liu2023cross,
  author = {Meng Liu and Kong Aik Lee and Longbiao Wang and Hanyi Zhang and Chang Zeng and Jianwu Dang},
  title = {{Cross-Modal Audio-Visual Co-Learning for Text-Independent Speaker Verification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=vwDm-nm7Bw},
}

Loading related content...

Figure