Abstract
Pre-trained Language Models (PLMs) often inherit sentiment biases from their training data, leading to spurious correlations where specific entities are incorrectly associated with fixed polarities. We present Gater, an adaptive framework that utilizes adversarial Gumbel attacks to mitigate this bias. By employing specialized expert networks and an Adaptive Training Unit (ATU), Gater dynamically injects sentiment noise during fine-tuning, forcing the model to prioritize contextual semantics over memorized entity-sentiment associations. Experimental results across seven benchmarks demonstrate that Gater effectively reduces bias while maintaining or improving classification accuracy.
Introduction
While Pre-trained Language Models (PLMs) like BERT and RoBERTa have revolutionized sentiment analysis, they are prone to "shortcut learning." During pre-training on massive corpora, these models may observe certain entities (e.g., "Apple") predominantly in positive contexts. This results in a sentiment bias, where the word embedding itself becomes inherently positive, regardless of the actual sentence context.
As shown in Figure 1, a traditional model might predict "Positive" for the neutral statement "Apple's apps are plentiful" simply due to the biased representation of "Apple." This lack of robustness hinders generalization to unbiased or out-of-distribution data. Existing debiasing techniques often suffer from "over-correction," where they remove too much semantic information, leading to a drop in overall performance.
Methodology: The Gater Framework
The authors propose Gater (Adaptive Gumbel-attacked classifier), which frames debiasing as an adversarial defense task. Instead of explicitly identifying and removing bias, Gater generates synthetic bias (noise) and trains the model to remain invariant to it.
Gumbel-Attack Experts
The framework utilizes multiple "expert" networks to generate sentiment-shifting noise. These experts employ the Gumbel distribution (specifically the Type-I generalized extreme value distribution). The advantage of Gumbel noise is its "semantic friendliness"—it can perturb the sentiment polarity of an embedding while preserving its core semantic meaning, unlike standard Gaussian noise which often renders embeddings unintelligible.
Adaptive Training Unit (ATU)
To prevent over-debiasing, Gater includes an Adaptive Training Unit (ATU). The ATU monitors the classifier's confidence levels during training. If the model exhibits high confidence or signs of bias for specific inputs, the ATU dynamically activates additional expert networks to provide more complex and diverse noise patterns. This ensures that the debiasing intensity is proportional to the difficulty of the input.
Multi-channel Parameter Updating
Gater employs a specialized parameter updating algorithm that distills knowledge from the experts. When an expert successfully identifies a bias, the classifier's weights are updated to "immunize" it against that specific pattern. To ensure diversity among experts, the system uses Mutual Information Minimization, forcing each expert to capture a distinct "flavor" of sentiment noise.
Experimental Evaluation
Gater was evaluated on seven diverse sentiment analysis benchmarks, covering binary classification (IMDb, SST-2, YELP-2, Amazon-2), multi-class classification (YELP-5, Amazon-5), and aspect-level sentiment (SemEval).
Key Results
- Improved Accuracy: Gater consistently outperformed vanilla PLMs and state-of-the-art debiasing methods. Notably, BERT+Gater achieved a 2.0% accuracy improvement over standard BERT on the SemEval dataset.
- Effective Bias Mitigation: In "sentiment shift" tests designed to probe hidden biases, Gater successfully shifted the sentiment scores of biased entities back toward neutrality compared to standard fine-tuning.
- Efficiency: Gater functions as an end-to-end framework during the fine-tuning phase and does not require additional domain-specific data. The expert networks are discarded after training, ensuring the final deployed model remains lightweight.
Discussion and Conclusion
Gater represents a significant step forward in addressing the shortcut problem in PLMs. By treating debiasing as an adversarial challenge and using adaptive noise injection, it avoids the pitfalls of over-correction. However, some limitations remain. The addition of expert networks increases training-time parameter overhead, and the ATU's confidence threshold requires careful hyperparameter tuning. Future research could explore extending the Gumbel-attack approach to other forms of social bias, such as gender or racial bias, to create more equitable and robust language models.
Summary updated