On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods

Kyle Boerstler, Vijay Keswani, Lok Chan, Jana Schaich Borg, Vincent Conitzer, Hoda Heidari, Walter Sinnott-Armstrong

Introduction

As artificial intelligence (AI) systems are increasingly deployed in high-stakes domains such as healthcare triage, autonomous driving, and financial lending, the challenge of "value alignment" becomes paramount. A common approach to encoding human ethics into AI is preference elicitation, where stakeholders participate in surveys—often involving pairwise comparisons—to define the system's underlying moral framework.

However, these elicitation methods typically rely on a "one-and-done" assumption: that human moral preferences are stable and consistent. A recent study, "On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods," challenges this premise. By evaluating participants' choices over multiple sessions, the researchers demonstrate that moral "ground truth" is often fluid, particularly in controversial scenarios.

Methodology

The researchers conducted two longitudinal studies focused on the ethical dilemma of kidney allocation. Participants were tasked with choosing which of two fictional patients should receive a transplant based on specific attributes.

Example of the pairwise comparison task used for kidney allocation preferences.

Study Design

  • Task Structure: Participants compared pairs of patients described by features such as age, number of dependents, and health habits.
  • Longitudinal Tracking: Unlike standard one-time surveys, participants completed 10 sessions over a two-week period.
  • Reliability Metrics: The study included "repeated scenarios"—identical patient pairs hidden within different sessions—to measure intra-rater reliability.
  • Modeling: The team employed Bradley-Terry models to quantify feature weights and machine learning models (Random Forests and Logistic Regression) to test the predictive power of data from a single session across the entire two-week span.

Empirical Findings

The study revealed significant instability in human moral judgment, especially when decisions were not clear-cut.

The Stability Gap

While participants were nearly 100% consistent on "uncontroversial" cases (e.g., choosing a patient who would gain 20 years of life over one who would gain none), their stability dropped significantly in "controversial" or "close-call" scenarios. In these instances, participants changed their minds 10–18% of the time.

Results from Study Two showing scenario feature values and corresponding response stability and reaction times.

Reaction Time and Indecision

A key finding was the correlation between reaction time (RT) and instability. Unstable responses were consistently associated with longer deliberation times. This suggests that in difficult moral cases, longer thinking does not necessarily lead to a more "correct" or stable answer; rather, it may indicate a decision boundary where the participant is essentially indifferent.

Predictive Decay

Models trained on a single session's data struggled to predict the same participant's future choices. Random Forest models achieved only 79–81% accuracy when predicting choices in subsequent sessions, highlighting the risk of over-fitting AI to a "snapshot" of human preference that may not hold true over time.

Discussion and Implications

The findings suggest that current "one-and-done" elicitation frameworks may result in AI systems that are misaligned with actual human values. If a person's "Monday preference" differs from their "Friday preference," which one should the AI follow?

Recommendations for AI Alignment

  1. Multi-Session Elicitation: Developers should move toward longitudinal data collection to capture the variance in human judgment.
  2. Accounting for Uncertainty: Elicitation interfaces should allow participants to express indifference or uncertainty, rather than forcing a binary choice in "toss-up" scenarios.
  3. Dynamic Modeling: AI systems must be designed to handle "noisy" human input, recognizing that moral preferences may be probabilistic rather than deterministic.

By acknowledging the inherent instability of human moral compasses, researchers can build more robust and truly aligned ethical AI systems.

Summary updated

Abstract

Preference elicitation frameworks feature heavily in the research on participatory ethical AI tools and provide a viable mechanism to enquire and incorporate the moral values of various stakeholders. As part of the elicitation process, surveys about moral preferences, opinions, and judgments are typically administered only once to each participant. This methodological practice is reasonable if participants’ responses are stable over time such that, all other things being held constant, their responses today will be the same as their responses to the same questions at a later time. However, we do not know how often that is the case. It is possible that participants’ true moral preferences change, are subject to temporary moods or whims, or are influenced by environmental factors we don’t track. If participants’ moral responses are unstable in such ways, it would raise important methodological and theoretical issues for how participants’ true moral preferences, opinions, and judgments can be ascertained. We address this possibility here by asking the same survey participants the same moral questions about which patient should receive a kidney when only one is available ten times in ten different sessions over two weeks, varying only presentation order across sessions. We measured how often participants gave different responses to simple (Study One) and more complicated (Study Two) controversial and uncontroversial repeated scenarios. On average, the fraction of times participants changed their responses to controversial scenarios (i.e., were unstable) was around 10-18% (±14-15%) across studies, and this instability is observed to have positive associations with response time and decision-making difficulty. We discuss the implications of these results for the efficacy of common moral preference elicitation methods, highlighting the role of response instability in potentially causing value misalignment between the stakeholders and AI tools trained on their moral judgments.
BibTeX
@inproceedings{boerstler2024stability,
  author = {Kyle Boerstler and Vijay Keswani and Lok Chan and Jana Schaich Borg and Vincent Conitzer and Hoda Heidari and Walter Sinnott-Armstrong},
  title = {{On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods}},
  booktitle = {AIES},
  year = {2024},
  doi = {10.1609/aies.v7i1.31626},
}
Figure