Fair Visual Recognition in Limited Data Regime using Self-Supervision and Self-Distillation

Pratik Mazumder, Pravendra Singh, Vinay P. Namboodiri

Abstract

Deep learning models generally learn the biases present in the training data. Researchers have proposed several approaches to mitigate such biases and make the model fair. Bias mitigation techniques assume that a sufficiently large number of training examples are present. However, we observe that if the training data is limited, then the effectiveness of bias mitigation methods is severely degraded. In this paper, we propose a novel approach to address this problem. Specifically, we adapt self-supervision and self-distillation to reduce the impact of biases on the model in this setting. Self-supervision and self-distillation are not used for bias mitigation. However, through this work, we demonstrate for the first time that these techniques are very effective in bias mitigation. We empirically show that our approach can significantly reduce the biases learned by the model. Further, we experimentally demonstrate that our approach is complementary to other bias mitigation strategies. Our approach significantly improves their performance and further reduces the model biases in the limited data regime. Specifically, on the L-CIFAR-10S skewed dataset, our approach significantly reduces the bias score of the baseline model by 78.22% and outperforms it in terms of accuracy by a significant absolute margin of 8.89%. It also significantly reduces the bias score for the state-of-the-art domain independent bias mitigation method by 59.26% and improves its performance by a significant absolute margin of 7.08%.

Introduction: The Challenge of Dataset Bias

Deep learning models are often "lazy" learners, finding the easiest path to minimize loss by latching onto spurious correlations. For instance, if a dataset of "nurses" is 90% female, a model might associate "woman" with "nurse" rather than learning the actual features of the profession. This phenomenon, known as dataset bias, is significantly amplified in "limited data regimes." When a model has very few examples to learn from, it clings even harder to biased shortcuts like background color or gender. Existing mitigation tools, such as adversarial training, often lose their effectiveness when data is scarce.

Methodology: Limited Data Bias Mitigation (LDBM)

The authors introduce the Limited Data Bias Mitigation (LDBM) approach. Instead of relying solely on traditional fairness constraints, LDBM leverages two techniques usually reserved for improving general performance: self-supervision and self-distillation.

Self-Supervised Auxiliary Task

LDBM utilizes SimSiam, a Siamese network architecture, to perform a self-supervised auxiliary task. The model is shown two different augmented views of the same image and tasked with recognizing that they represent the same underlying object. This forces the network to learn robust, meaningful representations that are invariant to "noise" (like color or rotation), making it harder for the model to rely on biased shortcuts.

SimSiam architecture used for the self-supervised auxiliary task in LDBM.

Self-Distillation

The second component is a two-stage self-distillation process. A "teacher" model is first trained using standard classification loss combined with the self-supervised task. Subsequently, a "student" model is trained to match the "soft" predictions (logits) of the teacher. This process acts as a regularizer, improving generalization and helping the model "forget" training-specific biases.

Experimental Setup

To evaluate LDBM, the authors created "limited" versions of popular fairness benchmarks by reducing training sets to just 5% of their original size while maintaining the original bias skew.

  • L-CIFAR-10S: A limited version of CIFAR-10 Skewed, where classes are heavily correlated with being either color or grayscale.
  • L-CelebA: A limited version of the CelebA face dataset, used to evaluate multi-label classification and gender bias.

Key Results and Insights

The core insight of this work is that bias is often a symptom of poor representation learning. By using self-supervision, the model is forced to learn what an object is regardless of its context.

Comparison of LDBM with various bias mitigation baselines on the L-CIFAR-10S dataset.

As shown in the results for the L-CIFAR-10S dataset:

  • Bias Reduction: LDBM reduced the bias score of the baseline model by 78.22%.
  • Accuracy Boost: It outperformed the baseline in terms of accuracy by an absolute margin of 8.89%.
  • Complementary Nature: When added to the state-of-the-art "Domain Independent" (DomInd) method, LDBM reduced its bias score by 59.26% and improved its accuracy by 7.08%.

Conclusion

This research demonstrates that self-supervision and self-distillation are highly effective for bias mitigation in data-scarce environments. While the method currently requires domain labels for evaluation, it provides a significant step toward building fair and robust visual recognition systems with minimal data.

Summary updated

Abstract

Deep learning models generally learn the biases present in the training data. Researchers have proposed several approaches to mitigate such biases and make the model fair. Bias mitigation techniques assume that a sufficiently large number of training examples are present. However, we observe that if the training data is limited, then the effectiveness of bias mitigation methods is severely degraded. In this paper, we propose a novel approach to address this problem. Specifically, we adapt self-supervision and self-distillation to reduce the impact of biases on the model in this setting. Self-supervision and self-distillation are not used for bias mitigation. However, through this work, we demonstrate for the first time that these techniques are very effective in bias mitigation. We empirically show that our approach can significantly reduce the biases learned by the model. Further, we experimentally demonstrate that our approach is complementary to other bias mitigation strategies. Our approach significantly improves their performance and further reduces the model biases in the limited data regime. Specifically, on the L-CIFAR-10S skewed dataset, our approach significantly reduces the bias score of the baseline model by 78.22% and outperforms it in terms of accuracy by a significant absolute margin of 8.89%. It also significantly reduces the bias score for the state-of-the-art domain independent bias mitigation method by 59.26% and improves its performance by a significant absolute margin of 7.08%.
BibTeX
@inproceedings{mazumder2022fair,
  author = {Pratik Mazumder and Pravendra Singh and Vinay P. Namboodiri},
  title = {{Fair Visual Recognition in Limited Data Regime using Self-Supervision and Self-Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00394},
}
Figure