Debiasing NLP Models Without Demographic Information

Hadas Orgad, Yonatan Belinkov

Abstract

Models trained on real-world data tend to imitate and amplify social biases. Common methods to mitigate biases require prior information on the types of biases that should be mitigated (e.g., gender or racial bias) and the social groups associated with each data sample. In this work, we introduce BLIND, a method for bias removal with no prior knowledge of the demographics in the dataset. While training a model on a downstream task, BLIND detects biased samples using an auxiliary model that predicts the main model's success, and down-weights those samples during the training process. Experiments with racial and gender biases in sentiment classification and occupation classification tasks demonstrate that BLIND mitigates social biases without relying on a costly demographic annotation process. Our method is competitive with other methods that require demographic information and sometimes even surpasses them.

Introduction

Most state-of-the-art debiasing techniques suffer from a significant "data tax": to remove bias, they require explicit demographic labels (e.g., race or gender) for every training instance. This requirement is problematic due to the high cost of manual annotation, privacy risks associated with collecting sensitive personal information, and the inherent complexity of human identity, which simple binary labels often fail to capture. This research introduces BLIND (Bias removaL wIth No Demographics), a method designed to build fairer models without any prior knowledge of the demographics in the training set.

Methodology: The BLIND Approach

BLIND builds upon the concept of Debiased Focal Loss (DFL). Traditional DFL uses an auxiliary model trained to predict demographic labels; samples that are easily identified by their demographics are down-weighted to prevent the model from relying on them.

In contrast, BLIND replaces the demographic predictor with a Success Detector. This auxiliary model attempts to predict whether the main model will succeed on a specific sample without seeing the actual task labels. The core intuition is that social bias is a form of "shortcut learning." If a simple auxiliary model can predict success, it is likely because the sample contains a shallow shortcut (like a gendered stereotype) rather than requiring deep linguistic understanding. By identifying and down-weighting these "easy" samples, BLIND forces the model to learn more robust, less biased features.

The BLIND architecture uses a Success Detector to identify biased shortcuts without requiring demographic labels during training.

Experimental Results

The authors evaluated BLIND on two benchmarks: sentiment analysis (using AAE and MUSE dialects as racial proxies) and occupation classification (using the "Bias in Bios" dataset).

Performance and Fairness

BLIND proved remarkably competitive against baselines that had access to full demographic data. In sentiment analysis, it significantly reduced the True Positive Rate (TPR) gap, nearly matching the performance of demographic-aware methods. In occupation classification, BLIND actually outperformed established methods like INLP and RLACE on certain fairness metrics, such as "Sufficiency," with only a minor (~3%) drop in overall accuracy.

Internal Unlearning

To understand why BLIND works, the researchers used Minimum Description Length (MDL) probing to measure how much demographic information remained in the model's internal layers. The results showed that BLIND makes demographic information harder to extract (indicated by lower compression rates in the probing task). This suggests that even without seeing gender or race labels, the model learned to discard these superficial correlations.

MDL probing results demonstrate that BLIND (represented in the 'with group labels' experiments) reduces the extractability of demographic information across both sentiment and occupation tasks.

Conclusion

BLIND represents a major step toward privacy-preserving fairness in NLP. By treating social bias as a robustness issue rather than a demographic labeling problem, it achieves competitive debiasing results without the ethical and financial costs of sensitive data collection. Future work will explore its application to long-form generation, non-binary identities, and morphologically rich languages where gender is more deeply embedded in grammar.

Summary updated

Abstract

Models trained on real-world data tend to imitate and amplify social biases. Common methods to mitigate biases require prior information on the types of biases that should be mitigated (e.g., gender or racial bias) and the social groups associated with each data sample. In this work, we introduce BLIND, a method for bias removal with no prior knowledge of the demographics in the dataset. While training a model on a downstream task, BLIND detects biased samples using an auxiliary model that predicts the main model's success, and down-weights those samples during the training process. Experiments with racial and gender biases in sentiment classification and occupation classification tasks demonstrate that BLIND mitigates social biases without relying on a costly demographic annotation process. Our method is competitive with other methods that require demographic information and sometimes even surpasses them.
BibTeX
@inproceedings{orgad2023debiasing,
  author = {Hadas Orgad and Yonatan Belinkov},
  title = {{Debiasing NLP Models Without Demographic Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=kaVYq3vswo},
}
Figure