Abstract
Discrimination can occur when the underlying unbiased labels are overwritten by an agent with potential bias, resulting in biased datasets that unfairly harm specific groups and cause classifiers to inherit these biases. In this paper, we demonstrate that despite only having access to the biased labels, it is possible to eliminate bias by filtering the fairest instances within the framework of confident learning. In the context of confident learning, low self-confidence usually indicates potential label errors; however, this is not always the case. Instances, particularly those from underrepresented groups, might exhibit low confidence scores for reasons other than labeling errors. To address this limitation, our approach employs truncation of the confidence score and extends the confidence interval of the probabilistic threshold. Additionally, we incorporate with co-teaching paradigm for providing a more robust and reliable selection of fair instances and effectively mitigating the adverse effects of biased labels. Through extensive experimentation and evaluation of various datasets, we demonstrate the efficacy of our approach in promoting fairness and reducing the impact of label bias in machine learning models.
Introduction
Machine learning models are fundamentally limited by the quality of their training data. When datasets contain "label bias"—where ground-truth labels are influenced by sensitive attributes like race or gender—models inevitably learn and perpetuate these discriminatory patterns. While many fairness-aware techniques focus on modifying model architectures or loss functions, fewer address the root cause: the biased data itself.
A common technique for cleaning data is "confident learning" (CL), which identifies and removes label errors by pruning examples where the model has low confidence. However, standard CL can be detrimental to fairness. Models often output lower confidence scores for underrepresented groups simply due to data scarcity. Standard CL mistakenly flags these as "errors" and discards them, inadvertently exacerbating dataset bias. This paper proposes a novel, model-agnostic framework that combines confident learning with a modified co-teaching paradigm and robust statistical thresholding to filter biased labels without harming minority representation.
Methodology
The proposed framework introduces two primary innovations to mitigate label bias while protecting valid data from minority groups:
1. Demographic Co-Teaching
Standard co-teaching involves training two models on the same data to cross-validate errors. The authors adapt this by training two identical models on separate demographic subsets (e.g., Model A on the privileged group, Model B on the disadvantaged group). Both models then evaluate the entire dataset. This cross-validation helps mitigate the "echo-chamber" effect of class imbalances and more effectively exposes labels biased against specific groups.
2. Truncated Confidence Thresholds
In standard CL, a probabilistic threshold (often the average confidence score of a class) is used to identify errors. However, models are frequently overconfident about majority groups, pulling the average threshold so high that minority data cannot pass. The authors replace this simple average with a robust statistical M-estimator (a truncation function). By mathematically reducing the influence of extremely high-confidence predictions, the threshold is lowered, giving low-confidence examples from underrepresented groups a better chance of being retained if they are correct.
Experimental Evaluation
The framework was evaluated by injecting symmetric and asymmetric bias into several benchmark datasets, including the Adult income dataset (sensitive attribute: gender), ProPublica COMPAS (race), Credit Loan Data (gender), and Law School Admissions (gender). A synthetic 2D dataset was also used for hyperparameter tuning.
Results and Discussion
The proposed method was compared against standard Confident Learning (CL), LongReMix, Label Correction (LC), and Group Peer Loss (GPL).
- Superior Fairness and Accuracy: The method consistently achieved the lowest fairness violations, measured by Difference of Equal Opportunity (DEO), while maintaining or improving predictive accuracy. For instance, on the Adult dataset with 20% bias, the error rate dropped from 22.61% (standard CL) to 15.75%, and the DEO improved from 0.17 to 0.12.
- Stability Under High Bias: The framework demonstrated remarkable stability. When injected bias was increased from 20% to 40%, baseline models showed significant spikes in error and fairness violations, whereas the proposed method exhibited only marginal degradation.
Conclusion
The core insight of this work is that low confidence in a model does not always equate to a label error; in a fairness context, it is often a symptom of underrepresentation. By using demographic co-teaching and robust thresholding, the proposed method cleans datasets in a model-agnostic way. While it currently requires access to sensitive attributes during data selection, it represents a significant step toward mitigating the inheritance of societal biases in machine learning.
Summary updated