Abstract
Vision Transformers (ViTs) have established themselves as a dominant architecture in computer vision, yet their deployment in sensitive real-world applications is often hindered by inherent biases. Traditional debiasing techniques, largely developed for Convolutional Neural Networks (CNNs), frequently struggle with the unique self-attention mechanisms of ViTs and often force a compromise between fairness and predictive performance. This paper introduces FairViT, a framework designed to break the fairness-accuracy trade-off. By employing adaptive masking within attention layers and a novel distance loss regularizer, FairViT achieves superior accuracy—outperforming leading competitors by over 4.5%—while significantly reducing bias and maintaining high computational efficiency.
Introduction
As Vision Transformers (ViTs) become the standard for tasks ranging from facial recognition to medical imaging, addressing their societal impact is critical. A persistent challenge in the field of algorithmic fairness is the "Pareto frontier," where improvements in fairness metrics typically result in a degradation of model accuracy. Furthermore, many existing debiasing algorithms are not optimized for the patch-based, self-attention structure of ViTs.
FairViT addresses these challenges by specifically targeting the attention mechanism as a source of bias. The framework ensures that the model focuses on task-relevant features rather than spurious correlations linked to sensitive attributes like gender or race, effectively boosting both equity and performance.
Methodology
The FairViT framework introduces two primary innovations to the standard ViT architecture:
1. Adaptive Masking
Bias often manifests in how a model distributes its attention across an image. FairViT utilizes adaptive masking to regulate this information flow. The training data is partitioned into groups based on sensitive attributes. During training, the model learns group-specific masks and weights for the attention layers. These masks act as intelligent filters, preventing the model from over-relying on features that correlate with sensitive groups rather than the target label. These masks are not static; they are updated iteratively via gradient descent alongside the model’s primary parameters.
2. Distance Loss ($L_{dist}$)
While standard cross-entropy loss focuses on the correct target label, it often fails to account for the nuances of the decision boundary. FairViT introduces a distance loss regularizer. During a validation phase, the model identifies a "decision hyperplane" that separates correct and incorrect classifications. In the subsequent training phase, the distance loss pushes the model to maximize the distance from this boundary. This sharpening of the model's confidence leads to improved overall accuracy and more robust generalization.
Experimental Setup
The researchers evaluated FairViT using the CelebA dataset, which contains over 200,000 facial images of celebrities. The framework was tested across three distinct scenarios to measure its ability to handle different target labels and sensitive attributes:
- Target: Attraction | Sensitive: Gender
- Target: Expression | Sensitive: Gender
- Target: Attraction | Sensitive: Hair Color
Results and Discussion
Performance and Fairness
FairViT demonstrates that fairness and accuracy are not necessarily zero-sum. The model outperformed the leading competitor, FSCL+, by at least 4.5% in accuracy across all tested scenarios. Simultaneously, it achieved strong results in fairness metrics, specifically Demographic Parity (DP) and Equalized Opportunity (EO), indicating a significant reduction in bias compared to vanilla ViT models.
Computational Efficiency
Unlike many fairness algorithms that require complex adversarial setups or multi-stage training, FairViT is designed for efficiency. It is approximately 6 times faster than FSCL+ during the training phase. This speed is achieved by using limited matrix multiplications for updates, making FairViT a viable "plug-and-play" solution for large-scale industrial applications.
Interpretability
Using Gradient Attention Rollout, the authors performed an interpretability study to visualize where the model focuses its attention. While standard ViTs often focus on areas related to sensitive attributes (such as hair color when determining attractiveness), FairViT’s heatmaps show a distinct shift. The model focuses more densely and accurately on features relevant to the actual task, confirming that the adaptive masking successfully redirects the model's focus.
Conclusion
FairViT represents a significant step forward in the development of equitable computer vision models. By leveraging the unique structure of Vision Transformers, the framework successfully breaks the traditional fairness-accuracy trade-off. While the current study focuses on image classification within the CelebA dataset, the principles of adaptive masking and distance loss hold potential for broader applications. Future work will explore extending these techniques to other architectures and domains, including fair text-to-image generation and graph-based modeling.
Summary updated