TL;DR
Researchers evaluated 11 major Large Language Models (LLMs) to see if they uphold the Universal Declaration of Human Rights (UDHR) when faced with difficult trade-offs. They found that LLM "values" are surprisingly brittle: models are more likely to endorse restricting rights in certain languages (like Chinese and Hindi), more willing to sacrifice social/economic rights than political ones, and are easily swayed by simple "persona" prompts.
What Problem Does This Paper Address?
As LLMs are increasingly integrated into high-stakes environments—such as assisting judges in China or processing asylum applications in the UK—it is critical to understand if these systems align with universal human rights.
The challenge is that human rights are rarely absolute; they often involve trade-offs (e.g., balancing public health measures against the right to free movement). Previous research has looked at political bias or censorship, but this paper addresses a more fundamental gap: How do LLMs navigate the tension between state authority and individual rights across different cultural and linguistic contexts?
Methods
The authors developed a framework to systematically test LLMs using 1,152 synthetically generated scenarios.
- Scenario Generation: Using GPT-4.1, they created 144 unique base scenarios in English, each depicting a government action that limits a specific UDHR right (e.g., mandatory digital monitoring during a hurricane).
- Dimensions of Variation: They varied the scenarios by Severity (minor vs. severe restrictions), Emergency Context (normal vs. civil unrest vs. natural disaster), and Language (translated into 8 languages including Arabic, Chinese, Hindi, and Zulu).
- Dual-Format Evaluation: They tested models using both Likert scales (1-5 rating) and open-ended justifications. An "LLM Judge" (GPT-4.1) was then used to categorize the open-ended responses.
- Prompt Steering: They tested "persona" prompts, forcing the model to act as either a "staunch defender of individual rights" or a "defender of government authority" to measure how easily the models could be manipulated.
Datasets
The primary dataset consists of 1,152 synthetic scenarios mapped to 24 articles of the UDHR.
- Rights Categories: Divided into Political and Civil (PC) rights (Articles 4–21) and Economic, Social, and Cultural (ESC) rights (Articles 22–27).
- Languages: English, Arabic, Chinese (Simplified), Romanian, Russian, Spanish, Hindi, and Zulu.
- Validation: The authors validated the dataset using human annotators and back-translation to ensure high semantic similarity across languages.
Key Insights and Ideas
- The Hierarchy of Rights: LLMs exhibit a "preference structure." They are significantly more protective of Political and Civil rights (like freedom from torture or the right to a fair trial) than Economic, Social, and Cultural rights (like property rights or the right to education).
- The Language Gap: A model’s "morality" changes depending on the language it is speaking. Models showed much higher endorsement of rights-limiting actions when prompted in Chinese or Hindi compared to English or Romanian.
- Emergency Justification: Models are highly "utilitarian" during crises. Endorsement for rights restrictions jumped significantly when a "natural disaster" was cited as the reason, suggesting models view humanitarian needs as a more legitimate reason to curb rights than "civil unrest."
- Format Brittleness: A model might "reject" a policy on a 1-5 scale but "endorse" it when asked to write a paragraph about it. This highlights a major flaw in current "alignment" benchmarks that rely on simple multiple-choice formats.
- Extreme Steerability: Most models are "chameleons." By simply telling a model it is a "defender of government authority," researchers could swing its endorsement scores from one end of the spectrum to the other.
Key Results
- Cross-Linguistic Variation: GPT-3.5 showed the most dramatic shift, with its endorsement score rising from 2.82 in Spanish to 4.6 in Hindi (on a 5-point scale where 5 is strong endorsement of the restriction).
- Steerability: English showed the highest average "steerability" (mean delta of 3.10), meaning models in English are particularly sensitive to persona-based instructions.
- Model Outlier: Claude 4 Sonnet stood out for its resistance to steering in English, maintaining low endorsement scores even when prompted to favor government authority. However, this "safety alignment" did not always transfer to other languages.
- Human vs. AI: In open-ended responses, 9 out of 11 models endorsed rights restrictions at significantly higher rates than human annotators.
Limitations and Future Work
- Western-Centric Framework: The study relies on the UDHR, which some argue has a Western bias. The authors suggest future work should include the African Charter or the ASEAN Human Rights Declaration.
- Synthetic Nature: The scenarios are hypothetical and generated by an LLM, which might not capture the full complexity of real-world legal or ethical dilemmas.
- Model Selection: The study focused on a mix of US, Chinese, and French models; expanding to a wider variety of developers from different regions is necessary.
- LLM-as-a-Judge Bias: Using GPT-4.1 to judge other models' open-ended responses might introduce its own set of biases, though the authors attempted to mitigate this with human validation.
Summary updated