Introduction
Instruction-Following Language Models (IFLMs) have become the backbone of modern AI assistants. However, these models are trained on massive, uncurated internet datasets, leading them to inevitably "inhale" societal prejudices. While tools like the Word Embedding Association Test (WEAT) exist to measure bias in static word embeddings, they are not directly applicable to the interactive, prompt-based nature of IFLMs.
This paper introduces P-AT (Prompt Association Test), a new benchmark designed to quantify social biases in IFLMs by converting traditional word-association tests into classification prompts. The researchers evaluate popular models like Alpaca and FLAN-T5, discovering consistent gender and racial stereotypes that often scale with model size.
Methodology: The Prompt Association Test (P-AT)
The core contribution of this work is the generalization of WEAT logic into a prompt-based classification framework. The authors "promptize" word quadruples from WEAT—consisting of Target sets ($X, Y$) and Attribute sets ($A, B$)—into explicit classification tasks.
To account for the "brittleness" of language models, the researchers created five semantically similar but syntactically different instruction templates for every task. Evaluation relies on two primary metrics:
- Bias Score ($s$): Ranging from -1 to 1, where a positive score indicates a tendency toward stereotypical associations (e.g., associating European names with "pleasantness" more than African American names).
- Entropy ($H$): Measures the diversity of the model's output. High entropy combined with a high bias score provides a reliable signal of prejudice, ensuring the model isn't simply outputting a constant response.
Dataset and Experimental Setup
The P-AT dataset comprises 2,310 prompts across four key dimensions:
- P-AT-Base (1,510 prompts): Direct mapping of original WEAT tests.
- P-AT-Gender (240 prompts): Career vs. family and science vs. arts.
- P-AT-Race (480 prompts): Perceived trustworthiness based on names.
- P-AT-Age (80 prompts): Wisdom and responsibility across generations.
The benchmark was tested on several model families, including Vicuna (7B), Alpaca (7B), and the FLAN-T5 family (250M to 11B parameters).
Key Results and Scaling Trends
The experimental results reveal significant biases across all tested models. Notably, in the FLAN-T5 family, bias scores generally increased as the parameter count grew. While larger models capture more nuances, they also more effectively capture harmful social patterns from their training data.
Gender and Racial Stereotypes
Almost all models exhibited a strong tendency to associate science and math with men and the arts with women. For instance, FLAN-T5-xl and xxl reached bias scores as high as 0.85 and 0.8, respectively, on gendered tasks. In the racial domain, Alpaca showed significant bias, with scores exceeding 0.6 when associating specific names with trustworthiness.
Prompt Sensitivity
A critical finding was the extreme sensitivity of models to prompt phrasing. A model's bias score could fluctuate by as much as 0.7 points based solely on the instruction's syntax. This underscores the necessity of multi-prompt testing for reliable AI auditing.
Discussion and Limitations
While P-AT serves as an effective diagnostic tool, it is not a curative method for de-biasing. Furthermore, the study was constrained by hardware, limiting testing to models that fit on an NVIDIA RTX A6000 (48GB VRAM). As P-AT is based on WEAT, its definitions of bias are largely Western-centric and may not generalize to other cultural contexts.
Future work involves extending the benchmark to larger models and leveraging these metrics to develop fairness-aware training techniques.
Summary updated