Introduction
As artificial intelligence (AI) systems are increasingly deployed in high-stakes domains such as healthcare triage, autonomous driving, and financial lending, the challenge of "value alignment" becomes paramount. A common approach to encoding human ethics into AI is preference elicitation, where stakeholders participate in surveys—often involving pairwise comparisons—to define the system's underlying moral framework.
However, these elicitation methods typically rely on a "one-and-done" assumption: that human moral preferences are stable and consistent. A recent study, "On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods," challenges this premise. By evaluating participants' choices over multiple sessions, the researchers demonstrate that moral "ground truth" is often fluid, particularly in controversial scenarios.
Methodology
The researchers conducted two longitudinal studies focused on the ethical dilemma of kidney allocation. Participants were tasked with choosing which of two fictional patients should receive a transplant based on specific attributes.
Study Design
- Task Structure: Participants compared pairs of patients described by features such as age, number of dependents, and health habits.
- Longitudinal Tracking: Unlike standard one-time surveys, participants completed 10 sessions over a two-week period.
- Reliability Metrics: The study included "repeated scenarios"—identical patient pairs hidden within different sessions—to measure intra-rater reliability.
- Modeling: The team employed Bradley-Terry models to quantify feature weights and machine learning models (Random Forests and Logistic Regression) to test the predictive power of data from a single session across the entire two-week span.
Empirical Findings
The study revealed significant instability in human moral judgment, especially when decisions were not clear-cut.
The Stability Gap
While participants were nearly 100% consistent on "uncontroversial" cases (e.g., choosing a patient who would gain 20 years of life over one who would gain none), their stability dropped significantly in "controversial" or "close-call" scenarios. In these instances, participants changed their minds 10–18% of the time.
Reaction Time and Indecision
A key finding was the correlation between reaction time (RT) and instability. Unstable responses were consistently associated with longer deliberation times. This suggests that in difficult moral cases, longer thinking does not necessarily lead to a more "correct" or stable answer; rather, it may indicate a decision boundary where the participant is essentially indifferent.
Predictive Decay
Models trained on a single session's data struggled to predict the same participant's future choices. Random Forest models achieved only 79–81% accuracy when predicting choices in subsequent sessions, highlighting the risk of over-fitting AI to a "snapshot" of human preference that may not hold true over time.
Discussion and Implications
The findings suggest that current "one-and-done" elicitation frameworks may result in AI systems that are misaligned with actual human values. If a person's "Monday preference" differs from their "Friday preference," which one should the AI follow?
Recommendations for AI Alignment
- Multi-Session Elicitation: Developers should move toward longitudinal data collection to capture the variance in human judgment.
- Accounting for Uncertainty: Elicitation interfaces should allow participants to express indifference or uncertainty, rather than forcing a binary choice in "toss-up" scenarios.
- Dynamic Modeling: AI systems must be designed to handle "noisy" human input, recognizing that moral preferences may be probabilistic rather than deterministic.
By acknowledging the inherent instability of human moral compasses, researchers can build more robust and truly aligned ethical AI systems.
Summary updated