FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs

Sepehr Dehdashtian, Lan Wang, Vishnu Boddeti

Introduction

Large-scale Vision-Language Models (VLMs), such as CLIP, have significantly advanced zero-shot classification capabilities. However, these models often inherit and amplify biases present in their massive, uncurated training datasets. These biases manifest as spurious correlations—such as associating specific professions with gender—or as challenges with intrinsic dependencies where target attributes are naturally correlated with sensitive ones.

FairerCLIP introduces a novel framework designed to mitigate these biases by debiasing CLIP’s representations within Reproducing Kernel Hilbert Spaces (RKHSs). This approach offers a unified, efficient, and flexible solution that operates effectively across both supervised and unsupervised settings, providing a significant step toward safer real-world AI deployment.

Methodology

The core innovation of FairerCLIP lies in its mathematical formulation, moving away from traditional neural network layers toward functions in RKHSs. This shift allows the model to treat debiasing as a representation-alignment problem.

Dependence Measure and Objective

The framework utilizes a variation of the Hilbert-Schmidt Independence Criterion (HSIC) to quantify the statistical dependence between model representations and sensitive attributes (e.g., race or gender). The optimization objective is three-fold:

  1. Bias Reduction: Minimize the information related to the sensitive attribute ($S$).
  2. Utility Preservation: Maintain information relevant to the target task ($Y$).
  3. Cross-Modal Alignment: Ensure image and text representations remain aligned with each other.

Optimization via Closed-Form Solvers

Unlike existing methods that rely on iterative backpropagation, FairerCLIP leverages kernel methods to solve a generalized eigenvalue problem. This allows for alternating closed-form solvers, resulting in a significant computational speedup—typically 4x to 10x faster than gradient-based alternatives like Contrastive Adapter. This "advanced linear algebra" approach provides a universal approximation capability that rivals shallow MLPs while being much faster to converge.

Flexibility and Pseudo-Labeling

FairerCLIP is uniquely versatile. In scenarios where ground-truth labels are unavailable, the framework utilizes CLIP’s initial zero-shot predictions as pseudo-labels. These labels are iteratively refined during the debiasing process, allowing the model to function effectively in a fully unsupervised capacity.

Experimental Evaluation

The authors evaluated FairerCLIP across several benchmarks to test fairness, robustness, and efficiency:

  • Waterbirds: Focused on spurious correlations between foreground objects (birds) and backgrounds.
  • CelebA: Tested both spurious correlations (e.g., blonde hair vs. sex) and intrinsic dependencies (e.g., high cheekbones vs. sex).
  • FairFace: Measured demographic parity and skew across seven different race groups.
  • Chicago Face Database (CFD): A high-quality, small-scale dataset used to assess sample efficiency.

Key Insights and Results

Efficiency and Speed

FairerCLIP demonstrates remarkable training efficiency, achieving 4x to 10x speedups over state-of-the-art methods. Additionally, it requires fewer parameters than MLP-based debiasing modules, making it a lightweight addition to existing CLIP pipelines.

Fairness and Robustness

On the CelebA intrinsic dependency task, FairerCLIP reduced the Equal Opportunity Difference (EOD) to a negligible 0.002% for ResNet-50 models while maintaining high classification accuracy. In the Waterbirds dataset, it achieved a Worst-Group (WG) accuracy of 86.0%, outperforming several supervised baselines even when operating in its unsupervised mode.

Sample Efficiency

The framework excels in data-constrained environments. On the CFD dataset, which contains only approximately 600 samples, FairerCLIP maintained high accuracy and low bias, whereas most baseline methods failed to show significant fairness improvements.

Limitations and Future Work

While FairerCLIP is highly effective, the authors identify a few areas for further research. In extremely data-poor environments, heavy data augmentation may be necessary to accurately estimate dependence measures. Furthermore, while the study primarily utilized the RBF (Gaussian) kernel, exploring alternative kernel types might yield better performance for specific data distributions. Finally, the method's reliance on initial zero-shot predictions in unsupervised settings means that extreme initial bias could potentially act as a bottleneck for iterative refinement.

Conclusion

FairerCLIP represents a significant advancement in the development of fair and robust Vision-Language Models. By leveraging the universal approximation capabilities of kernel methods and the efficiency of closed-form optimization, it provides a scalable, lightweight tool for "scrubbing" biases from representations without the need for massive retraining.

Summary updated

Abstract

Large pre-trained vision-language models such as CLIP provide compact and general-purpose representations of text and images that are demonstrably effective across multiple downstream zero-shot prediction tasks. However, owing to the nature of their training process, these models have the potential to 1) propagate or amplify societal biases in the training data and 2) learn to rely on spurious features. This paper proposes FairerCLIP, a general approach for making zero-shot predictions of CLIP more fair and robust to spurious correlations. We formulate the problem of jointly debiasing CLIP’s image and text representations in reproducing kernel Hilbert spaces (RKHSs), which affords multiple benefits: 1) Flexibility: Unlike existing approaches, which are specialized to either learn with or without ground-truth labels, FairerCLIP is adaptable to learning in both scenarios. 2) Ease of Optimization: FairerCLIP lends itself to an iterative optimization involving closed-form solvers, which leads to 4×-10× faster training than the existing methods. 3) Sample Efficiency: Under sample-limited conditions, FairerCLIP significantly outperforms baselines when they fail entirely. And, 4) Performance: Empirically, FairerCLIP achieves appreciable accuracy gains on benchmark fairness and spurious correlation datasets over their respective baselines.
BibTeX
@inproceedings{dehdashtian2024fairerclip,
  author = {Sepehr Dehdashtian and Lan Wang and Vishnu Boddeti},
  title = {{FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=HXoq9EqR9e},
}
Figure