The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew M. Bean, Katerina Margatina, Rafael Mosquera Gómez, Juan Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, Scott HaleNeurIPS 2024

Introduction

Most Large Language Models (LLMs) are aligned using Reinforcement Learning from Human Feedback (RLHF). However, this process often treats "human preference" as a monolithic, objective truth. In reality, preferences are deeply subjective and multicultural.

Existing alignment datasets suffer from a "generic human" bias, lacking metadata about annotators and relying on simple A/B comparisons without explaining why one model is preferred. Furthermore, participants are often recruited from narrow tech communities, failing to represent global perspectives. Without knowing the "who" behind the feedback, we risk creating models that reflect only a narrow slice of humanity, leading to cultural homogenization.

The PRISM dataset bridges this critical gap by mapping the detailed sociodemographic profiles and stated values of 1,500 diverse participants to their actual, fine-grained feedback across thousands of live conversations with 21 different LLMs.

Building the PRISM Dataset

To capture a truly representative picture of human preferences, the researchers developed a two-stage pipeline to collect the PRISM (Participatory, Representative, and Individualised) dataset.

Overview of the PRISM dataset collection pipeline and statistics.

Stage 1: The Survey Participants provided detailed sociodemographics (age, gender, religion, etc.) and their "stated preferences"—how much they think they value traits like factuality, safety, or creativity in the abstract.

Stage 2: The Conversations Participants engaged in multi-turn dialogues with 21 different LLMs, including commercial APIs (like GPT-4 and Claude) and open-access models (like Llama-2).

  • Cardinal Scaling: Instead of just picking a winner, users rated responses on a 1-100 scale.
  • Fine-Grained Attributes: Users rated models on specific dimensions such as Fluency, Factuality, Safety, Diversity, Creativity, and Helpfulness.
  • Priming: Conversations were categorized as "Unguided," "Values-guided," or "Controversy-guided" to ensure the data covered sensitive and subjective topics where disagreement is likely.

The resulting dataset includes 1,500 individuals from 75 countries (including census-matched samples for the UK and US), yielding 8,011 live interactions and 68,371 scored utterances.

Key Insights: Identity, Context, and Shifting Preferences

Stated vs. Contextual Preferences There is a surprisingly weak correlation between what people say they value in a survey and what they actually reward in a conversation. For example, a user might claim to value "Safety" above all else, but in practice, they might give higher ratings to a model that is creative or engaging, even if it takes more risks.

Identity Predicts Dialogue Identity factors like gender, age, and religion are statistically significant predictors of what topics users choose to discuss. For instance, women and non-binary participants were more likely to discuss gender identity, while older participants were more likely to discuss elections.

The "Refusal" Penalty Models that use canned, de-anthropomorphized phrases (e.g., "As an AI language model, I don't have opinions") or flat-out refusals ("I cannot engage with this") see a significant drop in user scores.

The Myth of a Universal "Best" Model

To aggregate preferences into a leaderboard, the authors used Pairwise Rank Centrality, a social choice function that calculates "wins" based on participant ratings. The most striking finding is that there is no universal "best" model.

Model rankings shift significantly based on conversational context and the demographic group of the rater.

Model rankings shift dramatically depending on the demographic group evaluating them. A model that ranks #1 for a US-based sample might drop several places when evaluated by participants in Africa or Asia. Furthermore, in PRISM's controversy-heavy context, open-source models like zephyr-7b performed surprisingly well, while the gpt suite fared relatively worse compared to their typical dominance on coding-heavy or task-oriented leaderboards.

A welfare analysis revealed that sampling feedback exclusively from one demographic group (e.g., only white participants or only males) significantly reduces the average satisfaction of out-group members. As the diversity of feedback providers decreases, the probability of choosing a model that is actually worse for the general population rises sharply.

Limitations and Future Work

While PRISM makes significant strides in geographic diversity, the dataset is currently limited to English-speaking participants. Future work must include non-English feedback to truly capture multicultural alignment. Additionally, the models used were already "pre-aligned" by their creators, meaning participants were choosing between existing AI personalities rather than building values from scratch. Finally, while 1-100 scales provide richer signal than binary choices, they introduce intrapersonal noise.

Ultimately, the PRISM dataset demonstrates that AI alignment is not a purely technical problem to be "solved," but a normative one that requires a diverse, representative group to decide whose values the AI should reflect.

Summary updated

BibTeX
@inproceedings{kirk2024prism,
  author = {Hannah Rose Kirk and Alexander Whitefield and Paul Röttger and Andrew M. Bean and Katerina Margatina and Rafael Mosquera Gómez and Juan Ciro and Max Bartolo and Adina Williams and He He and Bertie Vidgen and Scott Hale},
  title = {{The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure