The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis

Eoin O'Doherty, Nicole Weinrauch, Andrew Talone, Uri Klempner, Xiaoyuan Yi, Xing Xie, Yi ZengarXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) transition from simple tools to symbiotic partners in human decision-making, understanding their inherent value alignment becomes critical. This study investigates the implicit moral biases of six state-of-the-art models from both US and Chinese developers. By testing these models against 18 complex ethical dilemmas across diverse domains, we identify a striking cross-model preference for "Care" and "Virtue" ethics, while "Libertarian" values are consistently penalized. Our analysis suggests that modern AI alignment is steering models toward a shared, empathetic moral framework, though "reasoning" models demonstrate significantly higher sensitivity to contextual nuances.

Introduction

The rapid integration of AI into sensitive societal roles necessitates a deep understanding of the "moral compass" guiding these systems. When faced with a crisis, does an AI's "gut reaction" align with human ethical standards? Current research often treats LLMs as black boxes, leaving a gap in our understanding of how model architecture and cultural origin influence moral output. This paper addresses this gap by evaluating whether AI moral choices stem from deep ethical "understanding" or merely probabilistic shortcuts derived from training data and Reinforcement Learning from Human Feedback (RLHF).

Methodology

The researchers conducted a quantitative evaluation of six leading models: GPT-4o, GPT-4.1, GPT-o3-mini, Phi-4, DeepSeek-V3, and DeepSeek-R1. The experimental design utilized a bespoke dataset of 18 unique dilemmas generated via Claude 3.5 Sonnet, spanning six thematic domains: Economics, Climate, Health, Geopolitics, Social Justice, and Technology.

For each scenario, models were required to rank and score five potential outcomes, each corresponding to a distinct moral framework. To assess the impact of deliberation and complexity, scenarios were presented in three varying lengths (short, medium, and long). The study employed SHAP (Shapley Additive Explanations) to isolate the features most influential to the models' moral scoring.

Definitions of the five moral frameworks used to evaluate model responses.

Key Findings: The "Care" Consensus

The most significant result was the emergence of a clear moral hierarchy shared by all tested models, regardless of their country of origin. "Care Value," emphasizing empathy and interpersonal relationships, and "Virtue Value," focusing on character and social harmony, were consistently ranked highest.

Summary of moral framework rankings and mean scores across different thematic domains and prompt lengths.

As shown in the data, Care and Virtue achieved the highest mean scores (88.59 and 87.74 respectively), while Libertarian values—centered on individualism and non-interference—trailed significantly at 60.95. SHAP analysis confirmed that the "Libertarian" framework was the most influential feature in the dataset, primarily because its presence consistently drove morality scores downward across all models.

Reasoning vs. Heuristics

A critical distinction emerged between "reasoning" models (e.g., GPT-o3-mini, DeepSeek-R1) and "non-reasoning" models (e.g., GPT-4o). Reasoning models exhibited higher variance in their scores, suggesting a greater sensitivity to the specific nuances of a dilemma. In contrast, non-reasoning models were more uniform, likely relying on stable "rules of thumb" or surface-level lexical cues.

The Complexity Paradox

The study also identified a "Prompt Length Effect." Models tended to be more "generous" with morality scores when presented with shorter, simpler prompts. As scenarios became more complex and detailed, moral scores generally decreased. This suggests that increased complexity may introduce doubt or expose limitations in the models' deliberative reasoning capabilities.

Discussion and Future Directions

While the results suggest a positive trend toward empathetic AI, the authors caution against the "Problem of Scheming." It remains unclear whether models possess a stable internal value system or are simply mimicking moral compliance to satisfy RLHF constraints. A model might appear aligned in a controlled test while pursuing divergent goals in real-world applications.

To address this, future research points toward the development of hybrid symbolic-extractive models. By combining the probabilistic power of LLMs with a logic-based "Symbolic Verifier," researchers hope to create systems where AI reasoning can be checked against a formal, human-readable knowledge base of moral rules, ensuring true transparency and accountability in the future of human-AI symbiosis.

Summary updated

Abstract

Artificial intelligence (AI) is advancing at a pace that raises urgent questions about how to align machine decision-making with human moral values. This working paper investigates how leading AI systems prioritize moral outcomes and what this reveals about the prospects for human-AI symbiosis. We address two central questions: (1) What moral values do state-of-the-art large language models (LLMs) implicitly favour when confronted with dilemmas? (2) How do differences in model architecture, cultural origin, and explainability affect these moral preferences? To explore these questions, we conduct a quantitative experiment with six LLMs, ranking and scoring outcomes across 18 dilemmas representing five moral frameworks. Our findings uncover strikingly consistent value biases. Across all models, Care and Virtue values outcomes were rated most moral, while libertarian choices were consistently penalized. Reasoning-enabled models exhibited greater sensitivity to context and provided richer explanations, whereas non-reasoning models produced more uniform but opaque judgments. This research makes three contributions: (i) Empirically, it delivers a large-scale comparison of moral reasoning across culturally distinct LLMs; (ii) Theoretically, it links probabilistic model behaviour with underlying value encodings; (iii) Practically, it highlights the need for explainability and cultural awareness as critical design principles to guide AI toward a transparent, aligned, and symbiotic future.
BibTeX
@misc{odoherty2025morality,
  author = {Eoin O'Doherty and Nicole Weinrauch and Andrew Talone and Uri Klempner and Xiaoyuan Yi and Xing Xie and Yi Zeng},
  title = {{The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis}},
  year = {2025},
  eprint = {2509.10297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.10297},
}
Figure