Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting

Ruixi Lin, Hwee Tou NgACL 2023 · OpenReview

Abstract

Language models exhibit systematic error patterns that mirror human cognitive biases, such as availability bias and the framing effect. This paper introduces novel metrics to quantify these biases in BERT-based prompting, specifically within the domain of Drug-Drug Interaction (DDI) extraction. Our findings reveal that models are biased toward classes with high input variance, even when training data is balanced. Furthermore, we demonstrate that the "tone" of a prompt—the framing effect—can be leveraged to significantly improve model performance, with "toned-down" language increasing $F_1$ scores from 10.2% to 44.6%.

Introduction

As prompt-based learning becomes the standard interface for language models, their sensitivity to prompt construction has become a critical area of study. This research moves beyond identifying random errors to quantifying "bias modes" that resemble human cognitive shortcuts. Specifically, the authors investigate:

  1. Availability Bias: The tendency of a model to predict a "majority" class because it is more "available" or easier to recall due to high input variety.
  2. Framing Effect: The phenomenon where the presentation of information (e.g., definitive vs. hedged tone) alters the model's decision.

Methodology

The study utilizes a BERT-base model fine-tuned using ADAPET for Drug-Drug Interaction (DDI) extraction. To quantify biases, two metrics were developed:

  • Availability Bias Score: Measured using "dummy prompts" (where drug info is replaced by "N/A"). A perfectly unbiased model would guess uniformly (20% per class); deviations from this baseline indicate bias.
  • Framing Effect Paraphrase (FEP) Score: An extension of BERTScore that rewards the inclusion of "softer" terms (e.g., "may," "reportedly") in GPT-3 generated paraphrases.

Quantifying Availability Bias

A key insight of this work is that "majority class" status is driven by input variance as much as sample size. In the DDI dataset, the "Negative" class (no interaction) comprises over 85% of the data and covers a vast range of diverse non-interactions.

Availability bias: Confusion matrix showing misclassifications favoring the high-variance Negative class.

Even when trained on a perfectly balanced dataset (2,000 samples per class), the model maintained a bias toward the "Negative" class. On the full training set, the model showed a 52% bias score toward the "Negative" class. This suggests that the internal representation of a class is strengthened by the diversity of examples it encompasses, making it more "available" for prediction during inference.

The Framing Effect and Results

The researchers found that the specific framing of a prompt—its "tone"—drastically alters performance. By using GPT-3 to paraphrase drug descriptions into "toned-down" versions using hedging language, they observed a significant improvement in accuracy.

Comparison of F1 scores between original sentences and toned-down paraphrases.

On a specific subset of the test data, the $F_1$ score jumped from 10.2% to 44.6% simply by changing the framing. For prompts with high FEP scores (> 0.99), the probability of correcting a previously wrong prediction was a staggering 98.7%. This demonstrates that the framing effect is a highly predictable lever for improving model performance.

Discussion and Limitations

While the results are promising, the study notes limitations in paraphrase quality, as GPT-3 occasionally hallucinates or misses drug mentions. Additionally, the FEP score currently relies on a manually curated dictionary of hedging terms. Future work should explore the universality of these biases across other domains like legal or financial analysis and automate the discovery of framing rules.

Summary updated

BibTeX
@inproceedings{lin2023mind,
  author = {Ruixi Lin and Hwee Tou Ng},
  title = {{Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=5iMvH6gyDu},
}
Figure