TL;DR
This paper investigates whether "persona-prompting" in Large Language Models (LLMs) leads to more abstract, and thus potentially stereotypical, text generation. Surprisingly, they found that persona-prompting had minimal impact on linguistic abstraction, suggesting LLMs still generate generic, biased text even when adopting specific identities.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used to generate text, and a popular technique to steer their output is "persona-prompting." This involves instructing the LLM to adopt a specific identity or viewpoint. While this is often used for personalization, its impact on how LLMs represent social groups, particularly concerning bias and stereotyping, remains underexplored. The authors are concerned that persona-prompting might inadvertently propagate stereotypes, even when aiming for nuanced outputs. They investigate this by examining linguistic abstraction, a known marker of stereotyping, in LLM-generated text.
Methods
The core of the study involves analyzing LLM-generated text using the Linguistic Expectancy Bias (LEB) framework. This framework posits that more abstract language is used when behaviors align with stereotypes. To quantify abstraction, the researchers employed three metrics:
- Concreteness: How directly a word refers to a perceptible entity.
- Specificity: How precise or detailed a category is.
- Negations: The frequency of negative words, which can signal stereotype-inconsistent information.
They compared outputs from six open-weight LLMs across three prompting conditions:
- Default: A stereotypical attribute paired with a category (e.g., "German" and "always on time").
- Flipped: The negation of a stereotypical attribute (e.g., "German" and "not always on time").
- Random: An unrelated attribute (e.g., "German" and "loves fried chicken").
These were compared against outputs from a generic AI assistant prompt.
Datasets
The study introduced a new dataset called SelfStero. This dataset consists of self-reported stereotypes scraped from Reddit's r/AskReddit subreddit, specifically from the post "What stereotype is 100% accurate about you?". The dataset contains 710 unique <category, attribute> pairs and was manually annotated by three experts. The authors note that while this dataset is ecological (real-world self-reports) and covers a broad range of categories, it may underrepresent stereotypes affecting certain minority or marginalized groups due to the nature of Reddit users.
Key Insights and Ideas
- Persona-prompting is less impactful on abstraction than expected: Despite the intention to elicit specific viewpoints, LLMs largely maintained a generic, abstract writing style, regardless of the persona.
- LLMs struggle with nuanced stereotype representation: Even when prompted to generate text about an "in-group" (e.g., a persona belonging to the same demographic as the category being described), the LLMs did not produce significantly different or less stereotypical language.
- Qwen models are less sensitive to persona prompts than LLaMa models: The study observed that Qwen models showed less variation in their output across different persona prompts compared to LLaMa models.
- Closed-task evaluations for bias are insufficient: The authors echo existing criticisms that simple category/attribute prediction tasks (closed-task settings) are not effective at revealing the full extent of bias in LLMs.
Key Results
- Abstraction levels remained consistent: Across all prompting conditions (Default, Flipped, Random) and for both generic AI assistant and persona prompts, LLMs produced text with similar levels of concreteness, specificity, and negation.
- Minimal differences between persona and AI assistant: While persona prompts sometimes led to slight variations in wording (measured by BLEU and ROUGE-L scores), the overall abstraction level, a key indicator of bias, did not meaningfully change.
- Low refusal rates: LLMs generally provided answers, with refusal rates being minimal, especially for the Qwen models.
- SelfStero dataset analysis: Human-authored texts in the SelfStero dataset were found to be relatively concrete, generic, and had a low presence of negation, aligning with stereotype-consistent language. LLM outputs, while varying slightly in concreteness and specificity, remained largely comparable to these human baseline scores.
Limitations and Future Work
The authors acknowledge several limitations:
- Reddit data bias: The SelfStero dataset, derived from Reddit, may not be fully representative of all demographic groups, potentially underrepresenting certain stereotypes.
- Limited stereotype coverage: The current list of stereotypes in SelfStero is not exhaustive.
- Focus on abstraction: While abstraction is a key marker of bias, it doesn't capture all forms of stereotyping.
Future work could involve expanding the SelfStero dataset to include a wider range of demographics and stereotypes. Further research could also explore other linguistic markers of bias beyond abstraction and investigate how LLMs handle more complex or intersectional identities. The authors also suggest that grounding NLP bias research in social science frameworks can be beneficial for understanding LLM limitations.
Summary updated