Abstract
Large Language Models (LLMs) typically undergo alignment to a single "universal" set of human preferences, often overlooking the subjective and culturally dependent nature of social equity. We present FAIR-PP, a synthetic dataset of over 238,000 preference records grounded in real-world social survey archetypes. By categorizing fairness across five perspective dimensions and 98 equity topics, we enable granular persona-based alignment. Our proposed Weighted Direct Preference Optimization (WDPO) leverages sample reweighting to emphasize unique persona traits, significantly improving alignment accuracy and distinctiveness compared to standard methods.
Introduction
Most current LLM alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), treat human preference as a monolith. The goal is usually a single, universal "helpful and harmless" response. However, social values—particularly those concerning equity and fairness—are deeply personal and vary significantly across demographics. Existing datasets often lack the granularity to handle specific social equity topics or ignore personalization entirely. Furthermore, collecting high-quality human feedback on sensitive social topics is prohibitively expensive. FAIR-PP addresses these gaps by providing a framework to generate diverse, persona-driven preference data reflecting real-world social divisions.
The FAIR-PP Framework and Dataset
The researchers developed a multi-step pipeline to create the FAIR-PP dataset, which consists of 34,089 unique survey questions and 238,623 personalized preference data points. The dataset covers 28 social groups and 98 equity topics, such as "Fair Rewards" and "Basic Essentials."
Each question is framed across five "Perspective Dimensions":
- Distribution: Meritocracy vs. Egalitarianism
- Justice: Procedural vs. Distributive
- Identity: Individualism vs. Collectivism
- Trade-off: Social Norm vs. Equity Concerns
- Governance: Moral Constraints vs. Legal Restrictions
Methodology: Persona Grounding and Weighted Alignment
The framework adopts seven representative "value-based segments" from UK social survey data (e.g., Progressive Activists, Backbone Conservatives), anonymized into universal profiles. Using GPT-4o-mini, the researchers simulated these personas to answer the questions.
A core algorithmic contribution of this work is the introduction of Weighted Direct Preference Optimization (WDPO) and Weighted Supervised Fine-Tuning (WSFT). To align a model with a specific persona, the researchers assign higher weights to samples where the target persona’s preference is unique compared to the other six. This approach forces the model to learn the distinctive traits of a persona rather than just commonalities shared across all groups.
Positioning LLMs in the Preference Space
An analysis using Jensen-Shannon (JS) distance reveals that mainstream LLMs often possess a "default" personality. When mapped into the preference space, models like Llama 3, Mistral, and Qwen naturally align most closely with "Persona 3" (Disengaged Battlers).
This finding suggests that current training data or standard alignment processes may inadvertently bake in a specific set of social values, highlighting the need for intentional personalization frameworks like FAIR-PP.
Key Results
Experimental results demonstrate that the proposed WDPO method achieves high alignment scores (0.98 similarity to the target persona). Crucially, it increases the "margin"—the distance from non-target personas—by up to 12% compared to vanilla DPO. Furthermore, models trained on the structured FAIR-PP data successfully generalized their assigned persona to new, complex narrative scenarios not seen during training.
Conclusion and Future Work
FAIR-PP provides a nuanced framework for evaluating and mitigating AI bias. While the current dataset is synthetic and based on UK-specific social research, it establishes a scalable method for aligning LLMs with diverse human values. Future work will focus on incorporating actual human responses to validate data quality and expanding the personas to reflect a broader range of global cultural contexts.
Summary updated