Introduction
Language models (LMs) frequently inherit societal biases from their training corpora. While various debiasing techniques exist, they often struggle with three major hurdles: bias reintroduction during fine-tuning, a reliance on expensive protected attribute labels (e.g., race or gender), and poor scalability across multiple attributes. This paper introduces ADPLVR (Adapter-based Class-wise Low Variance Regularization), a novel method designed to mitigate these biases during downstream task fine-tuning without requiring explicit labels for protected attributes.
Methodology: ADPLVR
ADPLVR leverages parameter-efficient fine-tuning through adapters—small trainable modules inserted into frozen model backbones like BERT or RoBERTa. The core innovation is Class-wise Low Variance Regularization (LVR). This approach treats bias as intra-class variance. For instance, in an occupation classification task, embeddings for "Doctor" might cluster by gender. By minimizing the variance within each task category, the model is forced to discard features causing internal spread—typically the protected attributes—while retaining task-relevant information.
To stabilize this process, the authors employ two key components:
- Moving Average (ω): Calculates class centers across training batches to provide a stable reference point.
- Center Loss (Lc): Ensures that while intra-class variance is minimized, the centers of different classes remain distinct to preserve classification performance.
Experimental Setup
The method was evaluated on three distinct document classification benchmarks:
- BIOS: Predicting occupations from biographies (Gender bias).
- FDCL18: Hate speech detection (Race/Dialect bias).
- PAN16: Mention detection in tweets (Multi-attribute setting: Age and Gender).
Explicit indicators of protected attributes were removed from the text to ensure the model addressed implicit representational bias rather than keyword matching.
Key Results
ADPLVR was compared against several baselines, including supervised methods that utilize attribute labels (e.g., Adversarial training - ADV, Maximum Mean Discrepancy - MMD).
As shown in the experimental results, ADPLVR often outperforms supervised baselines in removing sensitive information. For example, on the BIOS dataset with a BERT-BASE backbone, ADPLVR reduced the "probe" accuracy (a proxy for remaining bias) to 59.2%, surpassing the best supervised baseline (61.9%). Crucially, this debiasing occurs with minimal impact on primary task performance, often staying within 0.5% of standard biased models. In some cases, such as FDCL18, removing biased "noise" actually improved task accuracy.
Discussion and Limitations
ADPLVR demonstrates that label-free debiasing is not only possible but highly effective by treating bias as irrelevant intra-class variance. Its attribute-agnostic nature allows for the simultaneous mitigation of multiple biases without additional computational overhead or the need for specific word lists.
However, the study notes several limitations:
- Binary Assumptions: Datasets often simplify gender and race into binary categories, which does not reflect the full spectrum of human identity.
- Task Scope: The method has primarily been tested on classification tasks; its efficacy in generative tasks remains an open question.
- Architecture: Evaluation was limited to encoder-based models like BERT and RoBERTa. Future work could explore if this applies to decoder-only models (like GPT-style LLMs).
Summary updated