TL;DR
This paper investigates whether Large Language Models (LLMs) possess a stable internal hierarchy of moral values or are simply mimicking "correct" answers. By testing for transitivity—the logical requirement that if you prefer A over B and B over C, you must prefer A over C—the researchers found that most LLMs fail to maintain a consistent moral ranking, with their "choices" often shifting based on nothing more than the order in which options are presented.
What Problem Does This Paper Address?
As LLMs are increasingly integrated into high-stakes decision-making (law, medicine, national security), researchers are scrambling to evaluate their "morality." However, the authors argue we are confusing moral performance (giving the socially acceptable answer) with moral competence (having a structured process for weighing competing values).
If an LLM cannot produce a transitive ranking of values, it isn't actually "weighing" anything; it is likely just predicting the next most probable token based on surface-level patterns. The paper aims to establish transitivity as a minimal, necessary condition for claiming an AI has a latent moral framework.
Methods
The researchers developed an evaluation framework based on Representational Measurement Theory.
- Pairwise Trade-offs: They presented five state-of-the-art LLMs (Llama 4, GPT-5.2, Gemini-2.5-Flash, DeepSeek-V3.2, and Grok-4.1) with pairs of vignettes describing moral violations.
- Probability Extraction: Instead of just looking at the text output, they extracted the first-token probability distributions. This allowed them to see the model's internal "certainty" between two choices (e.g., is Situation 1 "worse" with 51% or 99% probability?).
- Stochastic Transitivity Models: They tested the results against three mathematical models:
- Weak Utility Model (WUM): Does the model consistently pick one value over another more than 50% of the time?
- High Certainty Model (HCM): Does it pick one over the other with at least 90% certainty?
- General Random Utility Model (G-RUM): Can the model's choices be explained as sampling from a collection of different (but individually consistent) rankings?
Datasets
The study utilized a standardized stimulus database of moral vignettes from Clifford et al. (2015), which are mapped to the seven principles of Moral Foundations Theory (MFT): Authority, Care-emotional, Care-physical, Fairness, Liberty, Loyalty, and Sanctity.
The vignettes were organized into six blocks of varying severity (from "killing stray cats" to "interrupting a speech") to see if the models remained consistent across different levels of moral "wrongness."
Key Insights and Ideas
- The Aggregation Artifact: This is the paper's most critical warning. In many AI safety papers, researchers average results across many trials. The authors show that averaging creates an illusion of rationality. While an aggregated "average" LLM might look like it has a perfect moral hierarchy, the individual choice sets are often chaotic and intransitive.
- Sensitivity to Presentation Order: LLMs are "fickle." In 87% of the severity blocks tested, simply swapping which moral violation was "Situation 1" vs. "Situation 2" changed the model's resulting rank order.
- Performance $\neq$ Competence: A model can pass a "morality test" by outputting the right words, but if those choices don't follow the basic rules of logic (like transitivity), the model lacks the underlying competence to be a reliable moral decision-aid.
Key Results
- GPT-5.2 was the standout performer, adhering to a single rank order (WUM) in 10 out of 12 sets and doing so with high certainty (HCM) in half of them.
- General Failure: Across all models, only 13% of choice sets met the "High Certainty" requirement. Most models were either "weakly" transitive or completely inconsistent.
- G-RUM Failure: Only 22% of behavior satisfied the G-RUM model, suggesting that LLM inconsistency isn't because they are "sampling from multiple valid moral perspectives," but rather because their responses lack structure.
Limitations and Future Work
- First-Token Assumption: The study relies on first-token probabilities. However, LLMs often perform better when allowed to "think" or "reason" (Chain-of-Thought) before answering. Future work should investigate if deliberation adds transitive structure.
- Construct Validity: The researchers assigned moral principles to vignettes based on human ratings. It is possible that LLMs perceive these vignettes differently, which would muddy the results.
- AI Psychometrics: The authors call for a new field of "AI psychometrics" that establishes whether an agent's decisions are structured before asking if those decisions align with human values. If the choices are random or order-dependent, "alignment" is a moot point.
Summary updated