On Measuring Social Biases in Prompt-Based Multi-Task Learning

Afra Feyza Akyürek, Sejin Paik, Muhammed Yusuf Kocyigit, Seda Akbiyik, Serife Leman Runyun, Derry Wijaya

Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts can generalize into novel forms of language and handle novel tasks. While prompt engineering often focuses on achieving superior performance, this research investigates whether the way an input is encoded affects the social biases promoted in outputs. Specifically, we study T0, a large-scale multi-task model, and compare two semantically equivalent input forms: Question Answering (QA) and Natural Language Inference (NLI).

The Impact of Task Format on Bias

In prompt-based learning, researchers typically optimize for accuracy. However, LLMs inherit social biases from their training data. This paper asks a critical question: Does the task format itself influence the level of bias? If the same information is presented as a "Who did X?" question (QA) versus a "Does sentence A imply sentence B?" statement (NLI), does the model's internal prejudice fluctuate?

Comparison of QA and NLI prompt formats and their resulting bias scores for the T0 model.

Methodology and Datasets

The study utilized T0pp (an 11-billion parameter model) to compare semantically equivalent inputs across two formats. To ensure a robust comparison, the researchers used two primary benchmarks:

  • BBQ (Bias Benchmark for QA): An existing benchmark covering 11 categories (gender, race, religion, etc.). The authors converted this into an NLI format (BBQ → NLI).
  • BBNLI (Bias Benchmark for NLI): A newly introduced dataset featuring 3,642 unique pairs across 16 subtopics, capturing harmful stereotypes and cognitive biases. This was also converted into a QA format (BBNLI → QA).
Examples of the BBQ dataset and its conversion to the NLI format (BBQ → NLI) in both ambiguous and disambiguated contexts.

Key Insights: Familiarity Breeds Bias

The quantitative gap between formats is striking. Bias scores range from -100 to 100 (where 0 is unbiased):

  • BBNLI Dataset: The average bias score in NLI format was 6.79, but jumped to 19.41 in QA format.
  • BBQ Dataset (Gender): The QA format showed a bias score of 43.59, while the NLI format dropped to 4.49.
  • BBQ Dataset (Religion): QA bias was 37.16, while NLI bias was 13.98.

The results suggest that the model is consistently more biased when using the QA format—a format it saw extensively during its multi-task training. When forced into the NLI format (a "novel" task for this model), it acted more neutrally. Simply changing the task structure had a massive impact, far outweighing minor changes in prompt wording.

Conclusion and Implications

This research proves that task format matters significantly in the manifestation of social bias. In QA, the model often feels "forced" to pick a name even when the correct answer is "Unknown," whereas in NLI, it is more likely to correctly identify that a premise does not entail a biased hypothesis.

Future work in building multi-task models should consider "holding out" popular task formats during training. If models are only tested on formats they were trained on, we may be overestimating their performance while underestimating their potential for social harm.

Summary updated

Abstract

Large language models trained on a mixture of NLP tasks that are converted into a text-to-text format using prompts, can generalize into novel forms of language and handle novel tasks. A large body of work within prompt engineering attempts to understand the effects of input forms and prompts in achieving superior performance. We consider an alternative measure and inquire whether the way in which an input is encoded affects social biases promoted in outputs. In this paper, we study T0, a large-scale multi-task text-to-text language model trained using prompt-based learning. We consider two different forms of semantically equivalent inputs: question-answer format and premise-hypothesis format. We use an existing bias benchmark for the former BBQ and create the first bias benchmark in natural language inference BBNLI with hand-written hypotheses while also converting each benchmark into the other form. The results on two benchmarks suggest that given two different formulations of essentially the same input, T0 conspicuously acts more biased in question answering form, which is seen during training, compared to premise-hypothesis form which is unlike its training examples. Code and data are released under https://github.com/feyzaakyurek/bbnli.
BibTeX
@inproceedings{akyurek2022measuring,
  author = {Afra Feyza Akyürek and Sejin Paik and Muhammed Yusuf Kocyigit and Seda Akbiyik and Serife Leman Runyun and Derry Wijaya},
  title = {{On Measuring Social Biases in Prompt-Based Multi-Task Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=2bZCtDlj4E},
}
Figure