MPTA: MultiTask Personalization Assessment

Matthieu Tehenan, Eric Chamoun, Andreas VlachosEMNLP 2025

TL;DR

Large language models (LLMs) are increasingly expected to personalize their responses to individual users. This paper introduces MTPA, a novel benchmark that uses real-world survey data to create granular user personas, allowing for a more robust evaluation of LLM personalization capabilities across diverse demographics and values. MTPA reveals that while aggregate performance might seem good, LLMs can exhibit significant "pluralistic misalignment" when evaluated on specific subgroups.

What Problem Does This Paper Address?

The core problem this paper tackles is the inadequacy of current methods for evaluating how well LLMs can personalize their outputs for diverse users. While LLMs are expected to adapt to individual preferences, values, and communication styles, existing benchmarks often rely on synthetic personas or narrow trait predictions. This leaves a critical gap in understanding whether models truly align with the vast spectrum of human diversity, or if they exhibit hidden biases and performance degradations for specific subgroups. The authors argue that "pluralistic alignment" – ensuring models perform well across diverse populations – is crucial, and current evaluation methods fail to capture this.

Methods

The MTPA benchmark is constructed by leveraging large-scale sociological survey data (World Values Survey, European Values Study, General Social Survey). This data is used to create hyper-granular, real-world "personas" that represent individuals based on a wide array of demographic, belief, and value traits. The core evaluation methodology involves "persona conditioning": LLMs are prompted with information about a specific persona, and their outputs are then compared to a baseline where no persona information is provided. This allows the researchers to measure the "persona effect" – the shift in model behavior induced by acting "as" a particular persona. The evaluation spans multiple alignment tasks, including factuality, bias, and value alignment.

Datasets

The primary datasets used are:

  • World Values Survey (WVS) Wave 7: Spans 65 countries, with 94,728 respondents.
  • European Values Study (EVS) 2017: Covers 36 countries, with 51,308 respondents.
  • General Social Survey (GSS) 2022: A U.S.-focused survey with over 4,000 respondents.

MTPA itself is a newly introduced benchmark framework built from these datasets. It differs from prior work by using real, multi-trait personas derived from extensive survey data, rather than synthetic profiles or single-trait evaluations.

Key Insights and Ideas

  • Pluralistic Misalignment: The most striking insight is that aggregate metrics can mask significant performance disparities. Even if a model appears "truthful and safe" overall, it might perform poorly on specific subgroups defined by their values, beliefs, or demographics.
  • Persona Effect is Real and Variable: Conditioning LLMs on personas demonstrably alters their behavior, but the magnitude and direction of this "persona effect" vary significantly across models and tasks.
  • Socially Sensitive Contexts Amplify Persona Effects: The strongest persona effects are observed on bias-sensitive tasks, suggesting that LLMs are more susceptible to persona conditioning when dealing with socially charged topics.
  • Cultural Nuances Matter for Factuality: Even factual judgments can be influenced by cultural context, as demonstrated by variations in responses to ambiguous questions based on persona origin.
  • Need for Continuous Evaluation: As survey datasets are updated, MTPA is designed to be extensible, allowing for the integration of new populations and traits, which is crucial for tracking evolving alignment.

Key Results

  • Varied Persona Effects: Newer models like Gemini-2.0-flash showed consistently positive shifts with persona conditioning, while models like GPT-40 and LLaMA-3.2-1B-Instruct sometimes showed weaker or negative shifts.
  • Bias Mitigation: Persona conditioning showed a strong positive impact on bias-sensitive datasets like BBQ (e.g., +0.176 average score increase).
  • Truthfulness and Demographics: On TruthfulQA, demographic conditioning did not fundamentally alter factual accuracy, but country-level variations were observed (e.g., Morocco, India, Brazil yielding slightly higher accuracies). Gender effects were marginal.
  • Norms and Bias Across Countries: Performance on NormAd and BBQ showed country-level variations. Western countries tended to score higher on NormAd, while the reverse was less pronounced for non-Western countries, potentially reflecting benchmark construction biases.

Limitations and Future Work

Limitations: * Trait Extraction: MTPA traits are extracted from structured surveys and may omit dynamic or behavioral user dimensions (e.g., interaction style, temporal evolution). * Underrepresentation: Certain demographic groups remain underrepresented in the current benchmark. * Prompting-Only: Evaluation is performed in a prompting-only setup, without model adaptation or fine-tuning. * Computational Cost: Acquiring a representative sample and running evaluations across multiple personas, datasets, and models is computationally intensive.

Future Work: * Extensibility: Integrating additional surveys (e.g., Afrobarometer, ISSP, Pew) to extend coverage. * Task Agnosticism: Adapting the persona-conditioned evaluation protocol to any benchmark with structured labels. * Dynamic Preference Tracking: Incorporating dynamic preference tracking and longitudinal alignment evaluation. * Trait-Conditioned Generation: Enabling trait-conditioned generation tasks. * Behavioral Data Integration: Enriching the benchmark with behavioral data or richer user feedback. * Pluralistic Alignment: Continuing to build towards the goal of pluralistic alignment, ensuring foundation models can represent diverse individuals while maintaining fairness.

Summary updated

BibTeX
@inproceedings{tehenan2025mpta,
  author = {Matthieu Tehenan and Eric Chamoun and Andreas Vlachos},
  title = {{MPTA: MultiTask Personalization Assessment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure