Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts

Apoorv Garg, Deval Srivastava, Zhiyang Xu, Lifu HuangarXiv 2022 · DOI · arXiv

Introduction

Large-scale pre-trained language models (PLMs) such as BERT and RoBERTa are known to inherit societal biases from their training corpora. While research has extensively documented biases related to gender, race, and religion, sentiment bias—the latent association of specific words with positive or negative polarity—remains less explored at the token level. When a model internally assigns sentiment to a neutral word (e.g., "systematic" or "speculation"), it can inadvertently skew downstream tasks like automated grading or market analysis. This paper addresses the challenge of detecting these hidden biases using novel prompt-based probing techniques.

Methodology

To extract latent sentiment associations, the researchers developed two distinct methods: the Sentiment Association Test (SAT) and the Sentiment Shift Test (SST).

Sentiment Association Test (SAT)

The SAT measures the strength of association between a target word and positive versus negative contexts. By appending a prompt like "It was [MASK]." to a review (e.g., "The movie was awesome"), the test evaluates whether the model is more likely to predict a specific target word in the masked position for positive reviews compared to negative ones.

Overview of the Sentiment Association Test (SAT) mechanism.

Sentiment Shift Test (SST)

The SST employs a perturbation-based approach to observe how a "neutral" word influences the model's overall sentiment prediction. In this test, a neutral word is appended $K$ times to a review. The researchers then measure the shift in the model's sentiment prediction for that review using a new metric, $q$, to quantify the strength of the bias. If the repeated addition of a word like "religious" causes a positive review to be classified as negative, that word is identified as possessing a negative sentiment bias.

Overview of the Sentiment Shift Test (SST) showing the impact of neutral word repetition.

Experimental Setup

The study utilized human-annotated lexicons, including the VADER lexicon for polar words and the MPQA opinion corpus for neutral words. To provide context, 4,000 reviews were sampled from IMDB, Amazon, Yelp, and SST-2. The primary models evaluated were RoBERTa-base and RoBERTa-large.

Key Findings

The "Neutral" Illusion

The results indicate that human perception of neutrality does not always align with model representations. Approximately 39.25% of the 400 words labeled as "neutral" in human lexicons showed statistically significant sentiment bias. Words such as "vanilla," "cheese," and "speculation" were frequently treated as negative by the models, while "shone," "dominant," and "clout" exhibited positive bias.

Fine-tuning as an Amplifier

A critical finding is that fine-tuning PLMs on sentiment classification tasks significantly amplifies existing biases. In RoBERTa-base, the number of biased words increased by over 124% following fine-tuning. This suggests that the process of adapting a model to a specific task can reinforce and expand latent token-level associations.

Consistency and Model Scale

The SAT and SST methods showed high agreement, particularly for positively biased words, indicating that these biases are stable features of the model's architecture. While larger models like RoBERTa-large exhibited different specific bias patterns, they remained equally susceptible to the amplification effects of fine-tuning.

Conclusion

This research highlights the prevalence of token-level sentiment bias in PLMs and the risks associated with standard fine-tuning procedures. Future work should focus on expanding these tests to a broader range of architectures, such as GPT-4, and developing robust de-biasing techniques to ensure that "neutral" words remain truly neutral in automated systems.

Summary updated

Abstract

Due to the superior performance, large-scale pre-trained language models (PLMs) have been widely adopted in many aspects of human society. However, we still lack effective tools to understand the potential bias embedded in the black-box models. Recent advances in prompt tuning show the possibility to explore the internal mechanism of the PLMs. In this work, we propose two token-level sentiment tests: Sentiment Association Test (SAT) and Sentiment Shift Test (SST) which utilize the prompt as a probe to detect the latent bias in the PLMs. Our experiments on the collection of sentiment datasets show that both SAT and SST can identify sentiment bias in PLMs and SST is able to quantify the bias. The results also suggest that fine-tuning can possibly augment the existing bias in PLMs.
BibTeX
@misc{garg2022identifying,
  author = {Apoorv Garg and Deval Srivastava and Zhiyang Xu and Lifu Huang},
  title = {{Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Prompts}},
  year = {2022},
  eprint = {2204.07289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07289},
}
Figure