Introduction
Deep learning models are notorious for latching onto spurious correlations, or biases, in training data. For example, if a dataset contains mostly female nurses, a model might learn to associate gender with the profession rather than actual medical features. While many bias mitigation strategies exist, they typically share a common Achilles' heel: they require massive amounts of data to be effective.
This paper, presented at WACV 2022, addresses the "Limited Data Regime," where data scarcity significantly amplifies algorithmic bias and causes traditional fairness techniques to fail. The authors propose LDBM (Limited Data Bias Mitigation), a framework that repurposes self-supervision and self-distillation to effectively regularize fairness in low-data scenarios.
The Problem: Bias Amplification
The authors identify two primary issues when training data is scarce:
- Bias Explosion: Biased correlations are reinforced more strongly in low-data settings. In the CIFAR-10S dataset, the bias score of a standard model jumps from 0.074 (full data) to 0.349 (limited data)—a massive increase in unfairness.
- Mitigation Failure: Existing fairness algorithms lose effectiveness because they lack enough samples to learn the underlying distribution of "protected attributes" like gender or skin tone.
Methodology: The LDBM Framework
LDBM does not rely on complex adversarial games. Instead, it uses two established techniques in a novel way to induce fairness:
1. Self-Supervised Auxiliary Task (SimSiam)
The model is trained on a secondary task: recognizing that two different augmentations (views) of the same image represent the same object. By forcing the network to learn features invariant to changes like color or cropping, the model is discouraged from latching onto simple biased cues (like background color) and encouraged to learn robust, semantic features.
2. Self-Distillation
The authors first train a "Teacher" network using the self-supervised task. They then train a "Student" network to match the Teacher's "soft" predictions (logits). This process acts as a powerful regularizer, improving generalization and smoothing out biased "peaks" in the prediction distribution.
The Workflow: * Step 1: Train a Teacher network with standard Cross-Entropy loss plus SimSiam loss. * Step 2: Train a Student network using Cross-Entropy, SimSiam, and a Distillation loss that forces the Student to mimic the Teacher.
Experimental Results
The authors created "Limited" versions of existing fairness benchmarks, such as L-CIFAR-10S (reduced to 5% of original size) and L-CelebA (focusing on gender bias in face classification).
Key Insights
- Bias Reduction: On the L-CIFAR-10S dataset, LDBM reduced the baseline bias score by 78.22%.
- Accuracy Boost: Unlike some fairness methods that trade off accuracy for equity, LDBM improved accuracy by 8.89% over the baseline because it helps the model learn better features.
- Complementary Strength: When combined with existing state-of-the-art fairness methods like Domain Independent training, LDBM acted as a "booster," reducing that method's bias score by an additional 59.26%.
Conclusion
A key takeaway of this research is that generalization is inherently linked to fairness. Techniques designed to help models generalize better to unseen data—like self-supervision and distillation—prevent the model from over-fitting to the biases of the training set. LDBM provides a robust path forward for maintaining ethical AI standards even when data is scarce.
Summary updated