Mitigating Bias with Words: Inducing Demographic Ambiguity in Face Recognition Templates by Text Encoding

Tahar Chettaoui, Naser Damer, Fadi BoutrosarXiv 2025 · arXiv

TL;DR

This paper introduces Unified Text-Image Embedding (UTIE), a novel post-hoc strategy that mitigates demographic bias in face recognition systems by inducing demographic ambiguity. By leveraging the shared embedding space of Vision-Language Models (VLMs), UTIE enriches facial embeddings with text-derived features from other demographic groups, successfully reducing bias metrics while maintaining or even improving face verification accuracy.

What Problem Does This Paper Address?

Modern face recognition (FR) systems frequently exhibit demographic bias, leading to significant disparities in verification accuracy across different racial, gender, and age groups. A primary cause of this bias is feature entanglement: facial embeddings encode both identity-relevant features and demographic-specific attributes. When demographic traits dominate the embedding space, the system struggles to distinguish between individuals within the same demographic group, leading to higher error rates.

While existing methods attempt to address this during training (e.g., via adversarial debiasing or balanced loss functions), they often require retraining models from scratch or fine-tuning on massive datasets. This paper tackles the challenge of mitigating bias directly within frozen, pre-trained Vision-Language Models (VLMs) without the need for costly fine-tuning.

Methods

The authors propose Unified Text-Image Embedding (UTIE), which exploits the cross-modal alignment of VLMs (like CLIP) to obscure demographic information post-hoc.

Overview of the UTIE framework for inducing demographic ambiguity.

The process operates as follows:

  1. Image Embedding Extraction: A face image is passed through the VLM's image encoder to obtain the baseline embedding ($I$).
  2. Zero-Shot Demographic Prediction: The text encoder generates embeddings ($Tr$) for a set of demographic classes using a standard prompt template (e.g., "A photo of a {label} person."). The model predicts the dominant demographic attribute ($\hat{i}$) by finding the text embedding with the highest cosine similarity to the image embedding.
  3. Complementary Text Averaging: To neutralize this dominant attribute, UTIE excludes the predicted text embedding and computes the average vector ($\bar{T}$) of all remaining demographic text embeddings.
  4. Embedding Fusion: This mean text vector is added to the original image embedding ($I' = I + \bar{T}$).

By injecting features from all other demographic groups, the final face template becomes more demographically neutral, shifting the embedding closer to an "unbiased region" where identity cues take precedence over demographic markers.

Datasets

The approach was evaluated on two widely used face recognition fairness benchmarks:

  • Racial Faces in the Wild (RFW): Contains four testing subsets corresponding to African, Asian, Caucasian, and Indian groups to evaluate racial bias.
  • Balanced Faces in the Wild (BFW): Provides eight subgroups balanced across both gender and race, allowing for intersectional bias evaluation.

Key Insights and Ideas

  • Fighting Bias with "Ambiguity" Rather Than Suppression: Instead of trying to strip away demographic information entirely (which can degrade identity features), UTIE dilutes the dominant demographic signal by introducing features from other groups.
  • Flattening the Similarity Distribution: As shown in Figure 1, baseline face embeddings strongly align with their true demographic text label. UTIE successfully flattens this distribution, spreading the similarity scores more evenly across all demographic classes.
Comparison of average cosine similarities showing how UTIE distributes demographic similarity more evenly.
  • The Counter-Concept Validation: To prove their hypothesis, the authors tested a counter-concept called IE+PTE, where they added the predicted demographic text embedding to the image. This intentionally amplified the demographic signal, which consistently worsened both bias metrics and verification accuracy, validating that demographic dominance harms FR performance.

Key Results

The authors evaluated UTIE across three state-of-the-art VLMs: CLIP, OpenCLIP, and SigLIP. The method consistently reduced bias across both datasets while preserving verification performance:

  • Racial Bias (RFW): For baseline CLIP, UTIE improved mean accuracy slightly from 72.20% to 72.25%, while reducing the Standard Deviation (STD) across groups from 4.81 to 4.46 and the Skewed Error Ratio (SER) from 1.50 to 1.45.
  • Gender Bias (BFW): On the BFW dataset, applying UTIE to CLIP reduced the gender-based STD from 2.72 to 2.58 and lowered the SER from 1.28 to 1.27 while maintaining comparable overall accuracy.
  • Consistency Across Models: This trend of lower STD and SER held true across OpenCLIP and SigLIP, demonstrating that inducing demographic ambiguity is a robust strategy across different VLM architectures.

Limitations and Future Work

The primary limitation of UTIE is its dependency on the zero-shot classification accuracy of the text encoder. If the VLM misclassifies the dominant demographic attribute of the face image in the first step, the wrong text embeddings will be averaged, potentially leading to suboptimal debiasing.

The authors suggest that future research should focus on:

  • Advanced prompt engineering and text ensembling methods to improve zero-shot demographic prediction consistency.
  • Investigating the potential impact of this text-induction strategy on privacy leakage (e.g., whether adding text embeddings makes templates more vulnerable to inversion attacks).

Summary updated

Abstract

Face recognition (FR) systems are often prone to demographic biases, partially due to the entanglement of demographic-specific information with identity-relevant features in facial embeddings. This bias is extremely critical in large multicultural cities, especially where biometrics play a major role in smart city infrastructure. The entanglement can cause demographic attributes to overshadow identity cues in the embedding space, resulting in disparities in verification performance across different demographic groups. To address this issue, we propose a novel strategy, Unified Text-Image Embedding (UTIE), which aims to induce demographic ambiguity in face embeddings by enriching them with information related to other demographic groups. This encourages face embeddings to emphasize identity-relevant features and thus promotes fairer verification performance across groups. UTIE leverages the zero-shot capabilities and cross-modal semantic alignment of Vision-Language Models (VLMs). Given that VLMs are naturally trained to align visual and textual representations, we enrich the facial embeddings of each demographic group with text-derived demographic features extracted from other demographic groups. This encourages a more neutral representation in terms of demographic attributes. We evaluate UTIE using three VLMs, CLIP, OpenCLIP, and SigLIP, on two widely used benchmarks, RFW and BFW, designed to assess bias in FR. Experimental results show that UTIE consistently reduces bias metrics while maintaining, or even improving in several cases, the face verification accuracy.
BibTeX
@misc{chettaoui2025mitigating,
  author = {Tahar Chettaoui and Naser Damer and Fadi Boutros},
  title = {{Mitigating Bias with Words: Inducing Demographic Ambiguity in Face Recognition Templates by Text Encoding}},
  year = {2025},
  eprint = {2512.08981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.08981},
}
Figure