Abstract
Deep learning models often inherit biases from large-scale pretraining datasets, leading to poor performance on underrepresented classes. Correcting these biases typically requires massive amounts of balanced data, which is frequently unavailable in specialized domains. This paper introduces a pipeline that leverages small, balanced datasets through regularized fine-tuning and ensemble learning to mitigate algorithmic bias. By training multiple "counter-biased" models and aggregating their outputs via logit summation, the proposed method significantly enhances performance on minority classes without compromising overall model accuracy.
Introduction
In modern artificial intelligence, the "bias in, bias out" paradigm remains a significant challenge. State-of-the-art models are typically trained on massive datasets that often contain hidden imbalances. While the standard solution involves gathering more diverse data, this is often prohibitively expensive or impossible. Transfer learning, while efficient, often causes models to inherit the prejudices of their pretrained starting points. This research addresses the challenge of achieving fairness using only a small, balanced dataset to "counter-bias" a potentially poisoned pretrained model.
Methodology
The proposed pipeline utilizes a multi-stage approach to neutralize inherited bias:
- Data Splitting for Counter-Bias: The small, balanced dataset is partitioned into subsets designed to emphasize classes where the original model underperforms, creating a set of "counter-biased" experts.
- Regularized Fine-Tuning: To prevent overfitting on the limited data and avoid catastrophic forgetting of useful pretrained features, a penalty term ($\lambda$) is added to the loss function. This regularization forces the model to balance new evidence with prior knowledge.
- Ensemble Learning: Predictions from the original biased model and the new counter-biased models are combined. The researchers found that summing raw output scores (logits) outperformed simple majority voting, as it preserves the intensity of model certainty.
- Knowledge Distillation: An attempt was made to compress the ensemble into a single student model. However, this resulted in a significant drop in accuracy, indicating that maintaining fairness during model compression is a non-trivial task.
Experimental Results
The methodology was evaluated on two benchmarks: CIFAR-10 (with simulated bias) and the HAM10000 medical dataset. In the CIFAR-10 experiments, bias was introduced by removing 90% to 95% of labels for "Ship" and "Truck" classes in the pretraining phase.
As shown in the results for the 10% label case, the initial biased model achieved an overall accuracy of 84.6% but struggled significantly with the "Ship" (53.56%) and "Truck" (53.81%) classes. The proposed ensemble approach maintained a comparable overall accuracy of 84.29% while boosting the accuracy of the underrepresented classes to approximately 70%.
In more extreme cases where only 5% of labels were available for the minority classes, the ensemble method demonstrated even more dramatic improvements, raising accuracy from roughly 20% to over 60%.
Discussion and Conclusion
A key insight from this work is the importance of the regularization parameter $\lambda$, which acts as a "leash" to ensure the model finds a middle ground between its original knowledge and new, balanced evidence. Furthermore, intentional counter-biasing—creating models biased in opposite directions—proved more effective than attempting to train a single perfect model.
Despite these successes, the failure of knowledge distillation remains a significant limitation. While the ensemble is fair and accurate, its computational weight makes it less suitable for resource-constrained environments. Future work will focus on methods to "bake" fairness into lightweight, single-model architectures without sacrificing the gains achieved through ensemble methods.
Summary updated