Do large language models rank moral values? A transitivity test of moral competence

Katelyn A. Wasson, Kiwon Song, James M. Jennings, Brittney Currie, Konstantina Sokratous, Clintin P. StoberarXiv 2026 · arXiv

TL;DR

This paper investigates whether Large Language Models (LLMs) possess a stable internal hierarchy of moral values or are simply mimicking "correct" answers. By testing for transitivity—the logical requirement that if you prefer A over B and B over C, you must prefer A over C—the researchers found that most LLMs fail to maintain a consistent moral ranking, with their "choices" often shifting based on nothing more than the order in which options are presented.

What Problem Does This Paper Address?

As LLMs are increasingly integrated into high-stakes decision-making (law, medicine, national security), researchers are scrambling to evaluate their "morality." However, the authors argue we are confusing moral performance (giving the socially acceptable answer) with moral competence (having a structured process for weighing competing values).

If an LLM cannot produce a transitive ranking of values, it isn't actually "weighing" anything; it is likely just predicting the next most probable token based on surface-level patterns. The paper aims to establish transitivity as a minimal, necessary condition for claiming an AI has a latent moral framework.

Methods

The researchers developed an evaluation framework based on Representational Measurement Theory.

  1. Pairwise Trade-offs: They presented five state-of-the-art LLMs (Llama 4, GPT-5.2, Gemini-2.5-Flash, DeepSeek-V3.2, and Grok-4.1) with pairs of vignettes describing moral violations.
  2. Probability Extraction: Instead of just looking at the text output, they extracted the first-token probability distributions. This allowed them to see the model's internal "certainty" between two choices (e.g., is Situation 1 "worse" with 51% or 99% probability?).
  3. Stochastic Transitivity Models: They tested the results against three mathematical models:
  4. Weak Utility Model (WUM): Does the model consistently pick one value over another more than 50% of the time?
  5. High Certainty Model (HCM): Does it pick one over the other with at least 90% certainty?
  6. General Random Utility Model (G-RUM): Can the model's choices be explained as sampling from a collection of different (but individually consistent) rankings?
Summary of experimental and analytic procedures

Datasets

The study utilized a standardized stimulus database of moral vignettes from Clifford et al. (2015), which are mapped to the seven principles of Moral Foundations Theory (MFT): Authority, Care-emotional, Care-physical, Fairness, Liberty, Loyalty, and Sanctity.

The vignettes were organized into six blocks of varying severity (from "killing stray cats" to "interrupting a speech") to see if the models remained consistent across different levels of moral "wrongness."

Key Insights and Ideas

  • The Aggregation Artifact: This is the paper's most critical warning. In many AI safety papers, researchers average results across many trials. The authors show that averaging creates an illusion of rationality. While an aggregated "average" LLM might look like it has a perfect moral hierarchy, the individual choice sets are often chaotic and intransitive.
  • Sensitivity to Presentation Order: LLMs are "fickle." In 87% of the severity blocks tested, simply swapping which moral violation was "Situation 1" vs. "Situation 2" changed the model's resulting rank order.
  • Performance $\neq$ Competence: A model can pass a "morality test" by outputting the right words, but if those choices don't follow the basic rules of logic (like transitivity), the model lacks the underlying competence to be a reliable moral decision-aid.

Key Results

  • GPT-5.2 was the standout performer, adhering to a single rank order (WUM) in 10 out of 12 sets and doing so with high certainty (HCM) in half of them.
  • General Failure: Across all models, only 13% of choice sets met the "High Certainty" requirement. Most models were either "weakly" transitive or completely inconsistent.
  • G-RUM Failure: Only 22% of behavior satisfied the G-RUM model, suggesting that LLM inconsistency isn't because they are "sampling from multiple valid moral perspectives," but rather because their responses lack structure.
Number of vignette sets satisfying each transitivity model by LLM

Limitations and Future Work

  • First-Token Assumption: The study relies on first-token probabilities. However, LLMs often perform better when allowed to "think" or "reason" (Chain-of-Thought) before answering. Future work should investigate if deliberation adds transitive structure.
  • Construct Validity: The researchers assigned moral principles to vignettes based on human ratings. It is possible that LLMs perceive these vignettes differently, which would muddy the results.
  • AI Psychometrics: The authors call for a new field of "AI psychometrics" that establishes whether an agent's decisions are structured before asking if those decisions align with human values. If the choices are random or order-dependent, "alignment" is a moot point.

Summary updated

Abstract

LLMs are increasingly used to support decisions that require moral consideration. While LLMs perform well on morality benchmarks from psychology and philosophy, producing appropriate outputs on these benchmarks (moral performance) does not establish that an LLM weighs competing moral values at stake in a decision (moral competence). We test a minimal, necessary condition for such weighing: that an ordinal ranking can represent LLM moral choices. Such a ranking requires transitivity. To assess transitivity, we presented LLMs with pairwise trade-offs among moral value violations in vignette choice sets and extracted resulting choice probabilities from first-token distributions. Tested against three stochastic transitive choice models, most choice sets admitted weak or unstable orderings. Aggregating an LLM's responses, a common practice in LLM moral evaluation, produced transitive structure that matched no individual choice set. Evaluations of AI morality should establish that choices are structured before asking whether they are aligned.
BibTeX
@misc{wasson2026do,
  author = {Katelyn A. Wasson and Kiwon Song and James M. Jennings and Brittney Currie and Konstantina Sokratous and Clintin P. Stober},
  title = {{Do large language models rank moral values? A transitivity test of moral competence}},
  year = {2026},
  eprint = {2611.10265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10265},
}
Figure