Using the Veil of Ignorance to align AI systems with principles of justice

Laura Weidinger, Kevin R. McKee, Richard Everett, Saffron Huang, Tina Zhu, Martin J. Chadwick, Christopher Summerfield, Iason GabrielProceedings of the National Academy of Sciences 2023 · DOI

TL;DR

This paper explores using John Rawls' "Veil of Ignorance" thought experiment to determine fair principles for governing AI systems. They found that participants reasoning behind the Veil of Ignorance consistently prioritized principles that benefit the worst-off, driven by fairness concerns rather than personal risk attitudes or political leanings. This approach also led to more robustly endorsed principles, suggesting the Veil of Ignorance could be a valuable tool for aligning AI with societal values.

What Problem Does This Paper Address?

As AI systems become increasingly integrated into our lives, a critical question arises: how do we ensure these systems align with human morality and values? Existing approaches to AI alignment often fall into two camps: "moral intuitionist" methods that gather public or expert intuitions, and "theory-led" approaches that apply established philosophical frameworks. However, moral intuitions can conflict, and theory-led approaches may lack broad societal buy-in. This paper proposes the Veil of Ignorance (VoI) as a novel, fair process for selecting AI governance principles, aiming to overcome the limitations of existing methods by ensuring impartiality and robust endorsement.

Methods

The researchers conducted five incentive-compatible studies with a total of 2,508 participants. The core experimental design involved a computer-based harvesting task where participants, acting as a group with an AI assistant, had to choose a principle to govern the AI's actions.

  • Veil of Ignorance (VoI) Condition: Participants chose a principle before knowing their position or advantage within the group.
  • Control Condition: Participants chose a principle after knowing their position and how it would affect them.

Participants were presented with two principles: a prioritarian principle (prioritizing the worst-off) and a maximization principle (maximizing overall group return). The studies varied in format, including descriptive tasks and an immersive, real-time harvesting game. To assess "reflective endorsement," participants were later asked if they would still endorse their initial choice after experiencing the outcome and knowing their position. Participant explanations for their choices were analyzed to understand the reasoning behind their decisions.

Datasets

Not discussed in the paper. The studies used custom-built platforms for online experiments and did not rely on pre-existing datasets for training or evaluation.

Key Insights and Ideas

  • Fairness as the Primary Driver: The most striking finding is that participants behind the VoI consistently favored the prioritarian principle, and this preference was primarily driven by an unprompted focus on fairness, not by individual risk aversion or political ideology. This suggests the VoI effectively elicits a sense of impartiality.
  • Robust Endorsement: Principles chosen behind the VoI were more likely to be endorsed even when participants later had a self-interested incentive to change their minds. This "reflective endorsement" indicates that VoI-derived principles are more stable and less susceptible to personal bias.
  • The Power of Language: The way principles are presented matters. When principles were described using clear, linguistic terms, the VoI effect was stronger. Abstract labels ("Principle A," "Principle B") weakened the VoI's influence, highlighting the role of language in shaping moral reasoning.
  • Beyond Risk Aversion: Contrary to some prior interpretations of the VoI, this study demonstrates that it's not solely about mitigating personal risk. The emphasis on fairness suggests a deeper, more altruistic motivation at play when individuals are placed in an impartial position.

Key Results

  • Principle Choice: Participants in the VoI condition were significantly more likely to choose the prioritarian principle over the maximization principle across multiple studies (e.g., Study 1: OR = 9.3, p = .003).
  • Fairness-Based Reasoning: Participants behind the VoI were significantly more likely to explain their choices in terms of fairness (e.g., Study 2: OR = 2.3, p < .001). This effect was stronger when participants believed they were interacting with other humans.
  • Reflective Endorsement: Participants who chose the prioritarian principle behind the VoI were more likely to stick with that choice when faced with a self-interested motivation to change (e.g., Study 1: OR = 8.6, p = .005).
  • Political Independence: Political orientation (liberal-conservative or left-right) showed a much weaker predictive power for principle choice compared to fairness considerations.

Limitations and Future Work

The authors acknowledge several areas for future research:

  • Broader AI Applications: While the studies used a harvesting game, future work should explore how the VoI applies to more complex and diverse AI domains like healthcare, education, and manufacturing.
  • More Principle Options: Testing a wider range of principles, including hybrid ones that combine prioritarian and maximization elements, could provide a more nuanced understanding.
  • Cross-Cultural Differences: The current studies were conducted online, and future research should investigate whether these findings hold across different cultural contexts.
  • Representation of Principles: Further exploration into how the linguistic and visual representation of principles impacts VoI reasoning is warranted.
  • AI Capabilities: The impact of varying AI capabilities on principle selection behind the VoI could be a fruitful area of investigation.

Overall, this paper presents a compelling case for the Veil of Ignorance as a practical and effective method for developing fair and robust principles to guide the development and deployment of AI systems.

Summary updated

Abstract

The philosopher John Rawls proposed the Veil of Ignorance (VoI) as a thought experiment to identify fair principles for governing a society. Here, we apply the VoI to an important governance domain: artificial intelligence (AI). In five incentive-compatible studies (<b><i>N</i></b> = 2, 508), including two preregistered protocols, participants choose principles to govern an Artificial Intelligence (AI) assistant from behind the veil: that is, without knowledge of their own relative position in the group. Compared to participants who have this information, we find a consistent preference for a principle that instructs the AI assistant to prioritize the worst-off. Neither risk attitudes nor political preferences adequately explain these choices. Instead, they appear to be driven by elevated concerns about fairness: Without prompting, participants who reason behind the VoI more frequently explain their choice in terms of fairness, compared to those in the Control condition. Moreover, we find initial support for the ability of the VoI to elicit more robust preferences: In the studies presented here, the VoI increases the likelihood of participants continuing to endorse their initial choice in a subsequent round where they know how they will be affected by the AI intervention and have a self-interested motivation to change their mind. These results emerge in both a descriptive and an immersive game. Our findings suggest that the VoI may be a suitable mechanism for selecting distributive principles to govern AI.
BibTeX
@inproceedings{weidinger2023using,
  author = {Laura Weidinger and Kevin R. McKee and Richard Everett and Saffron Huang and Tina Zhu and Martin J. Chadwick and Christopher Summerfield and Iason Gabriel},
  title = {{Using the Veil of Ignorance to align AI systems with principles of justice}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2023},
  doi = {10.1073/pnas.2213709120},
}
Figure