Text-Guided Face Recognition Using Multi-Granularity Cross-Modal Contrastive Learning

Md Mahedi Hasan, Shoaib Meraj Sami, Nasser NasrabadiWACV 2024

TL;DR

This paper introduces Text-Guided Face Recognition (TGFR), a novel framework that leverages natural language descriptions to improve face recognition performance, especially in challenging low-quality surveillance scenarios. By employing a multi-granularity cross-modal contrastive learning approach, TGFR effectively aligns textual and visual features, leading to significantly enhanced accuracy over existing state-of-the-art methods.

What Problem Does This Paper Address?

Traditional face recognition (FR) systems often struggle in real-world surveillance settings where images are frequently low-resolution, noisy, or occluded. While incorporating "soft biometrics" like facial marks or gender can help, this paper tackles the challenge of integrating richer, more descriptive information from natural language captions. The core problem is bridging the semantic gap between image and text modalities, especially given the inherent subjectivity and potential ambiguity in textual descriptions, to create a more robust and accurate FR system.

Methods

The proposed TGFR framework builds upon existing FR models (like AdaFace and ArcFace) and a pre-trained vision-language foundation model (similar to CLIP). The key innovations lie in two modules:

  • Face-Caption Alignment Module (FCAM): This module is designed to learn aligned global and local features from both image and text. It employs a suite of contrastive losses:
  • Caption-Image Contrastive Loss (CICL): Aligns global image and caption embeddings.
  • Word-Region Contrastive Loss (WRCL): Learns fine-grained alignments between image sub-regions and specific words in the caption, using an attention mechanism to weigh sub-regions.
  • Intra-Modality Contrastive Loss (IMCL): Provides self-supervision within each modality (image-image pairs and caption-caption pairs) to enhance feature discriminability.
  • Identity Loss: A standard cross-entropy loss to ensure features are discriminative for identity recognition.
  • Face-Caption Fusion Module (FCFM): This module further refines the interaction between modalities. It uses cross-attention to allow the model to focus on relevant image regions when processing specific words and employs self-attention to capture complex relationships.

The training is conducted in two steps: first, the foundation model and FCAM are trained, keeping the FR image encoder frozen; then, the FCFM is trained.

Datasets

The paper evaluates TGFR on three publicly available face-caption datasets:

  • Multi-Modal CelebA-HQ (MMCelebA)
  • Face2Text
  • CelebA-Dialog

These datasets contain pairs of face images and corresponding textual descriptions.

Key Insights and Ideas

  • Textual descriptions are powerful soft biometrics: The paper strongly demonstrates that natural language descriptions can significantly boost FR performance, especially in challenging conditions, outperforming traditional soft biometrics.
  • Multi-granularity alignment is crucial: Simply aligning global image and text features isn't enough. The success of TGFR hinges on aligning at different levels – global image-caption, and fine-grained word-to-image-region interactions.
  • Contrastive learning is key to bridging the semantic gap: The various contrastive losses (CICL, WRCL, IMCL) are instrumental in learning a shared, discriminative multimodal space and overcoming the inherent differences between visual and textual data.
  • Low-quality images benefit the most: The performance gains are most pronounced on degraded images, highlighting TGFR's practical utility in real-world surveillance scenarios.

Key Results

  • Significant improvements in verification rate (VR): TGFR achieved substantial improvements across all three datasets compared to baselines and even state-of-the-art FR models. For example, on MMCelebA, TGFR achieved a VR of 68.20% at TAR@FAR=1e-5, a 3.52% improvement over CGFR and a 12.17% improvement over AdaFace.
  • Enhanced identification accuracy: Rank-1 identification accuracy also saw significant gains, with TGFR surpassing CGFR by 13.54% on MMCelebA.
  • Robustness to image quality: TGFR demonstrated remarkable resilience to image degradation. For instance, on "very bad" quality images, TGFR achieved an EER score of 17.16%, significantly outperforming AdaFace and CGFR.

Limitations and Future Work

  • Foundation Model Choice: While TGFR performs well with CLIP-like models, the paper notes that BERT-based encoders, while capable, don't align as well with the FR task's objectives. Future work could explore more specialized text encoders.
  • Dataset Limitations: The paper acknowledges that existing face-caption datasets can have limited numbers of pairs, which is a general challenge in this domain due to annotation complexity.
  • Computational Cost: While not explicitly detailed as a limitation, the multi-stage training and complex alignment modules likely incur significant computational overhead.

The paper's success suggests future work could explore more sophisticated fusion mechanisms, investigate the impact of different caption styles or lengths, and potentially extend TGFR to other vision-language tasks beyond face recognition.

Summary updated

BibTeX
@inproceedings{hasan2024text,
  author = {Md Mahedi Hasan and Shoaib Meraj Sami and Nasser Nasrabadi},
  title = {{Text-Guided Face Recognition Using Multi-Granularity Cross-Modal Contrastive Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure