TL;DR
This paper introduces Unified Text-Image Embedding (UTIE), a novel post-hoc strategy that mitigates demographic bias in face recognition systems by inducing demographic ambiguity. By leveraging the shared embedding space of Vision-Language Models (VLMs), UTIE enriches facial embeddings with text-derived features from other demographic groups, successfully reducing bias metrics while maintaining or even improving face verification accuracy.
What Problem Does This Paper Address?
Modern face recognition (FR) systems frequently exhibit demographic bias, leading to significant disparities in verification accuracy across different racial, gender, and age groups. A primary cause of this bias is feature entanglement: facial embeddings encode both identity-relevant features and demographic-specific attributes. When demographic traits dominate the embedding space, the system struggles to distinguish between individuals within the same demographic group, leading to higher error rates.
While existing methods attempt to address this during training (e.g., via adversarial debiasing or balanced loss functions), they often require retraining models from scratch or fine-tuning on massive datasets. This paper tackles the challenge of mitigating bias directly within frozen, pre-trained Vision-Language Models (VLMs) without the need for costly fine-tuning.
Methods
The authors propose Unified Text-Image Embedding (UTIE), which exploits the cross-modal alignment of VLMs (like CLIP) to obscure demographic information post-hoc.
The process operates as follows:
- Image Embedding Extraction: A face image is passed through the VLM's image encoder to obtain the baseline embedding ($I$).
- Zero-Shot Demographic Prediction: The text encoder generates embeddings ($Tr$) for a set of demographic classes using a standard prompt template (e.g., "A photo of a {label} person."). The model predicts the dominant demographic attribute ($\hat{i}$) by finding the text embedding with the highest cosine similarity to the image embedding.
- Complementary Text Averaging: To neutralize this dominant attribute, UTIE excludes the predicted text embedding and computes the average vector ($\bar{T}$) of all remaining demographic text embeddings.
- Embedding Fusion: This mean text vector is added to the original image embedding ($I' = I + \bar{T}$).
By injecting features from all other demographic groups, the final face template becomes more demographically neutral, shifting the embedding closer to an "unbiased region" where identity cues take precedence over demographic markers.
Datasets
The approach was evaluated on two widely used face recognition fairness benchmarks:
- Racial Faces in the Wild (RFW): Contains four testing subsets corresponding to African, Asian, Caucasian, and Indian groups to evaluate racial bias.
- Balanced Faces in the Wild (BFW): Provides eight subgroups balanced across both gender and race, allowing for intersectional bias evaluation.
Key Insights and Ideas
- Fighting Bias with "Ambiguity" Rather Than Suppression: Instead of trying to strip away demographic information entirely (which can degrade identity features), UTIE dilutes the dominant demographic signal by introducing features from other groups.
- Flattening the Similarity Distribution: As shown in Figure 1, baseline face embeddings strongly align with their true demographic text label. UTIE successfully flattens this distribution, spreading the similarity scores more evenly across all demographic classes.
- The Counter-Concept Validation: To prove their hypothesis, the authors tested a counter-concept called IE+PTE, where they added the predicted demographic text embedding to the image. This intentionally amplified the demographic signal, which consistently worsened both bias metrics and verification accuracy, validating that demographic dominance harms FR performance.
Key Results
The authors evaluated UTIE across three state-of-the-art VLMs: CLIP, OpenCLIP, and SigLIP. The method consistently reduced bias across both datasets while preserving verification performance:
- Racial Bias (RFW): For baseline CLIP, UTIE improved mean accuracy slightly from 72.20% to 72.25%, while reducing the Standard Deviation (STD) across groups from 4.81 to 4.46 and the Skewed Error Ratio (SER) from 1.50 to 1.45.
- Gender Bias (BFW): On the BFW dataset, applying UTIE to CLIP reduced the gender-based STD from 2.72 to 2.58 and lowered the SER from 1.28 to 1.27 while maintaining comparable overall accuracy.
- Consistency Across Models: This trend of lower STD and SER held true across OpenCLIP and SigLIP, demonstrating that inducing demographic ambiguity is a robust strategy across different VLM architectures.
Limitations and Future Work
The primary limitation of UTIE is its dependency on the zero-shot classification accuracy of the text encoder. If the VLM misclassifies the dominant demographic attribute of the face image in the first step, the wrong text embeddings will be averaged, potentially leading to suboptimal debiasing.
The authors suggest that future research should focus on:
- Advanced prompt engineering and text ensembling methods to improve zero-shot demographic prediction consistency.
- Investigating the potential impact of this text-induction strategy on privacy leakage (e.g., whether adding text embeddings makes templates more vulnerable to inversion attacks).
Summary updated