Fair Visual Recognition in Limited Data Regime using Self-Supervision and Self-Distillation

Pratik Mazumder, Pravendra Singh, Vinay P. Namboodiri

Introduction

Deep learning models are notorious for latching onto spurious correlations, or biases, in training data. For example, if a dataset contains mostly female nurses, a model might learn to associate gender with the profession rather than actual medical features. While many bias mitigation strategies exist, they typically share a common Achilles' heel: they require massive amounts of data to be effective.

This paper, presented at WACV 2022, addresses the "Limited Data Regime," where data scarcity significantly amplifies algorithmic bias and causes traditional fairness techniques to fail. The authors propose LDBM (Limited Data Bias Mitigation), a framework that repurposes self-supervision and self-distillation to effectively regularize fairness in low-data scenarios.

The Problem: Bias Amplification

The authors identify two primary issues when training data is scarce:

  1. Bias Explosion: Biased correlations are reinforced more strongly in low-data settings. In the CIFAR-10S dataset, the bias score of a standard model jumps from 0.074 (full data) to 0.349 (limited data)—a massive increase in unfairness.
  2. Mitigation Failure: Existing fairness algorithms lose effectiveness because they lack enough samples to learn the underlying distribution of "protected attributes" like gender or skin tone.

Methodology: The LDBM Framework

LDBM does not rely on complex adversarial games. Instead, it uses two established techniques in a novel way to induce fairness:

1. Self-Supervised Auxiliary Task (SimSiam)

The model is trained on a secondary task: recognizing that two different augmentations (views) of the same image represent the same object. By forcing the network to learn features invariant to changes like color or cropping, the model is discouraged from latching onto simple biased cues (like background color) and encouraged to learn robust, semantic features.

The SimSiam architecture used for self-supervised feature learning.

2. Self-Distillation

The authors first train a "Teacher" network using the self-supervised task. They then train a "Student" network to match the Teacher's "soft" predictions (logits). This process acts as a powerful regularizer, improving generalization and smoothing out biased "peaks" in the prediction distribution.

The self-distillation process where a Student network mimics a pre-trained Teacher.

The Workflow: * Step 1: Train a Teacher network with standard Cross-Entropy loss plus SimSiam loss. * Step 2: Train a Student network using Cross-Entropy, SimSiam, and a Distillation loss that forces the Student to mimic the Teacher.

Experimental Results

The authors created "Limited" versions of existing fairness benchmarks, such as L-CIFAR-10S (reduced to 5% of original size) and L-CelebA (focusing on gender bias in face classification).

Key Insights

  • Bias Reduction: On the L-CIFAR-10S dataset, LDBM reduced the baseline bias score by 78.22%.
  • Accuracy Boost: Unlike some fairness methods that trade off accuracy for equity, LDBM improved accuracy by 8.89% over the baseline because it helps the model learn better features.
  • Complementary Strength: When combined with existing state-of-the-art fairness methods like Domain Independent training, LDBM acted as a "booster," reducing that method's bias score by an additional 59.26%.

Conclusion

A key takeaway of this research is that generalization is inherently linked to fairness. Techniques designed to help models generalize better to unseen data—like self-supervision and distillation—prevent the model from over-fitting to the biases of the training set. LDBM provides a robust path forward for maintaining ethical AI standards even when data is scarce.

Summary updated

Abstract

Deep learning models generally learn the biases present in the training data. Researchers have proposed several approaches to mitigate such biases and make the model fair. Bias mitigation techniques assume that a sufficiently large number of training examples are present. However, we observe that if the training data is limited, then the effectiveness of bias mitigation methods is severely degraded. In this paper, we propose a novel approach to address this problem. Specifically, we adapt self-supervision and self-distillation to reduce the impact of biases on the model in this setting. Self-supervision and self-distillation are not used for bias mitigation. However, through this work, we demonstrate for the first time that these techniques are very effective in bias mitigation. We empirically show that our approach can significantly reduce the biases learned by the model. Further, we experimentally demonstrate that our approach is complementary to other bias mitigation strategies. Our approach significantly improves their performance and further reduces the model biases in the limited data regime. Specifically, on the L-CIFAR-10S skewed dataset, our approach significantly reduces the bias score of the baseline model by 78.22% and outperforms it in terms of accuracy by a significant absolute margin of 8.89%. It also significantly reduces the bias score for the state-of-the-art domain independent bias mitigation method by 59.26% and improves its performance by a significant absolute margin of 7.08%.
BibTeX
@inproceedings{mazumder2022fair,
  author = {Pratik Mazumder and Pravendra Singh and Vinay P. Namboodiri},
  title = {{Fair Visual Recognition in Limited Data Regime using Self-Supervision and Self-Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00394},
}
Figure