Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models

Somayeh Ghanbarzadeh, Yan Huang, Hamid Palangi, Radames Cruz Moreno, Hamed Khanpour

Abstract

Pre-trained language models (PLMs) such as BERT and RoBERTa have become foundational to modern Natural Language Processing. However, these models frequently inherit and amplify societal biases present in their unmoderated training data. Traditional debiasing methods often require significant computational resources or result in a trade-off between fairness and task performance. This paper discusses Gender-tuning, a novel "plug-and-play" method that integrates debiasing directly into the standard fine-tuning process. By employing a joint loss function that combines task-specific classification with gender-word perturbation via Masked Language Modeling (MLM), Gender-tuning effectively reduces bias while simultaneously maintaining or improving downstream task accuracy.

1. Introduction

Pre-trained Language Models (PLMs) are the backbone of modern NLP, yet they suffer from a significant "garbage in, garbage out" problem. Because they are trained on massive, unmoderated internet corpora, they inherit and amplify societal biases, such as associating "he" with "engineer" and "she" with "homemaker."

Existing solutions generally fall into two categories: resource-intensive methods that require re-training on curated datasets, and performance-degrading methods that strip away useful semantic information during the debiasing process. Gender-tuning addresses these challenges by providing a computationally efficient tool that requires no extra data and preserves task performance.

2. Methodology

The core of Gender-tuning is the integration of two training objectives into a single fine-tuning step using a joint loss function.

2.1 Gender-word Perturbation

During each training batch, the system identifies gendered words (e.g., "he," "actress," "men") using a predefined list. It then masks these words and uses Masked Language Modeling (MLM) to predict replacements. This process "perturbs" the text, often replacing gendered terms with neutral ones or opposite-gender terms. An analysis of these replacements reveals five primary types:

  • Neutral (46%): e.g., replacing a gendered term with "people."
  • Identical (29%): Retaining the original word.
  • Convert-gender (17%): e.g., swapping "she" for "he."
  • Same-gender (7%): Replacing with a different word of the same gender.
  • Deleting (1%): Removing the token entirely.

2.2 Joint Loss Function

The model is trained to classify these perturbed examples using the original ground-truth labels. For instance, if the sentence "He is a great actor" is labeled as positive, the model must also predict "positive" for the perturbed version "She is a great actress." The model minimizes a weighted sum of the perturbation loss ($L_{perturb}$) and the classification loss ($L_{fine-tuning}$):

$$L = L_{fine-tuning} + \lambda L_{perturb}$$

This approach forces the model to decouple gendered terms from task-specific labels, effectively "unlearning" biased associations.

3. Experimental Setup

The authors evaluated Gender-tuning using three benchmarks from the GLUE suite: SST-2 (Sentiment Analysis), CoLA (Linguistic Acceptability), and QNLI (Natural Language Inference). Bias was measured using the Sentence Encoder Association Test (SEAT), which calculates the distance between target concepts (like "career" vs. "family") and gendered attributes. Feminine and masculine word lists from prior research were used to identify tokens for perturbation.

4. Key Results and Analysis

Gender-tuning was compared against two state-of-the-art baselines: SENT-DEBIAS and FairFil.

4.1 Bias Mitigation

In nearly every test case, Gender-tuning achieved the lowest average absolute effect size (e-size) on the SEAT metric. For BERT on the SST-2 task, the bias score dropped from 0.291 (Original) to 0.176 (Gender-tuning).

4.2 Performance Gains

Surprisingly, Gender-tuning often improved accuracy. On the CoLA dataset with RoBERTa, accuracy rose from 57.35% to 58.54%, whereas other debiasing methods often caused accuracy to dip. This suggests that the perturbation process may serve as a form of data augmentation or regularization.

4.3 Efficiency

Because the process occurs during standard fine-tuning, it requires no additional datasets or massive compute cycles beyond what a researcher would already expend fine-tuning a model.

5. Limitations and Future Work

The method currently relies on a predefined list of gendered words and cannot mitigate biases associated with words not on that list. Additionally, the study focused on English; languages with complex gendered morphology (like Spanish or German) may require more sophisticated perturbation strategies. Future work could involve automatically expanding word lists or applying similar logic to racial or religious bias.

6. Conclusion

Gender-tuning represents a significant advancement in practical NLP debiasing. By performing debiasing and task-specific learning simultaneously, the model learns to ignore gendered cues while focusing on the nuances of the downstream task. This "plug-and-play" method offers a scalable path toward more equitable AI without sacrificing performance.

Summary updated

BibTeX
@inproceedings{ghanbarzadeh2023gender,
  author = {Somayeh Ghanbarzadeh and Yan Huang and Hamid Palangi and Radames Cruz Moreno and Hamed Khanpour},
  title = {{Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.336},
}
Figure