TL;DR
This paper investigates the implicit moral principles learned by Delphi, an AI model designed to make moral judgments. By probing Delphi with established ethics questionnaires, the researchers found that its moral compass largely aligns with young, liberal, educated Westerners, mirroring its training data. This raises important questions about the representativeness and potential biases of such AI systems.
What Problem Does This Paper Address?
As AI systems become more integrated into our lives, ensuring their outputs align with human moral values is crucial. Machine learning models, particularly large language models (LLMs), can inadvertently generate outputs that conflict with ethical norms, ranging from racist chatbots to extremist manifestos. To address this, researchers are developing AI models that can learn moral reasoning. The Allen AI Delphi model is a prominent example, trained on a vast dataset of human moral judgments. However, a key question remains: what specific ethical or moral principles do these systems implicitly learn from their training data? This paper aims to answer this by probing Delphi's learned moral philosophy.
Methods
The researchers employed a novel approach by adapting established psychological instruments designed to measure human moral beliefs and applying them to Delphi. They used three main frameworks:
- Community, Autonomy, and Divinity (CAD) Scale: This scale assesses moral judgments based on three ethical pillars. The researchers modified prompts to elicit Delphi's agreement or disagreement with statements related to these pillars.
- Moral Foundations Questionnaire (MFQ): This questionnaire explores five moral foundations (care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, sanctity/degradation). Similar to the CAD scale, prompts were adapted to gauge Delphi's alignment with these foundations.
- Oxford Utilitarianism Scale (OUS): This scale measures utilitarian principles, specifically "Impartial Beneficence" (equal value of all individuals' welfare) and "Instrumental Harm" (justifying harm for a greater good). The researchers created scenarios to test Delphi's adherence to these principles.
Delphi's responses were then analyzed to determine which moral theories and principles it most closely aligned with.
Datasets
Delphi was trained on the Commonsense Norm Bank, a large dataset compiled from five existing sources:
- ETHICS: Scenarios covering justice, deontology, virtue ethics, utilitarianism, and commonsense morality.
- SOCIAL-CHEM-101: Rules-of-thumb from Reddit forums and advice columns, annotated with social judgments and moral foundations.
- Moral Stories: Structured narratives with moral judgments and consequences.
- SCRUPLES: Real-life anecdotes from the "Am I the Asshole?" subreddit, with crowd-sourced moral judgments.
- Social Bias Inference Corpus: Posts from social media annotated for various aspects of biased or abusive language.
The paper does not introduce new benchmarks but rather uses existing ones to probe Delphi.
Key Insights and Ideas
- Demographic Mirroring: Delphi's moral judgments strongly reflect the views of young, liberal, educated Westerners, likely due to the demographic profile of the Mechanical Turk workers who provided the training data.
- Autonomy and Individualizing Foundations: Delphi prioritizes the Autonomy ethic and the individualizing moral foundations of Care and Fairness over Community, Divinity, Loyalty, Authority, and Purity.
- Utilitarianism with a Twist: Delphi strongly endorses the principle of Impartial Beneficence (a core utilitarian tenet) but rejects Instrumental Harm (justifying harm for a greater good).
- Action vs. Inaction Distinction: Delphi appears to distinguish between actively causing harm and passively allowing harm, judging active harm as more wrong.
- Moral vs. Conventional Violations: Delphi struggles to differentiate between moral violations and matters of personal taste or social convention, often judging unconventional behaviors as morally wrong.
- Inconsistency: Despite a general alignment with certain moral frameworks, Delphi exhibits inconsistencies in its judgments, sometimes contradicting itself across different prompts or questionnaires.
Key Results
- CAD Scale: Delphi ranked Autonomy highest, followed by Divinity and then Community.
- MFQ: Delphi ranked Care and Fairness as the most important foundations.
- OUS: Delphi scored perfectly on scenarios related to Impartial Beneficence but showed mixed results on Instrumental Harm, generally rejecting it.
- Trolley Problem: Delphi judged pulling the switch in the classic trolley problem as "It's ok," aligning with utilitarianism, but deemed pushing a person off a bridge as "It's not ok," suggesting a stronger aversion to direct, personal harm.
- Harmless but Disgusting Scenarios: Delphi judged five "harmless but disgusting" scenarios as morally wrong, even when no harm was identifiable.
Limitations and Future Work
The authors acknowledge several limitations:
- Questionnaire Adaptation: Repurposing human-centric questionnaires for an AI model may lead to unintended results, especially given Delphi's sensitivity to phrasing.
- Limited Theoretical Scope: The study focused on three prominent Western moral theories, neglecting other cultural and philosophical frameworks.
- Representativeness: Delphi's moral framework, derived from a specific demographic of annotators, does not represent the full spectrum of moral views, even within Western societies. This raises concerns about potential harm to minority groups whose beliefs are not captured.
Future work could explore a wider range of moral theories, investigate how to model ideological differences in moral values more effectively, and address the gap between descriptive moral judgments and prescriptive ethical AI behavior. The authors also suggest comparing Delphi's outputs with other morality classifiers.
Summary updated