What Counts Underlying LLMs’ Moral Dilemma Judgments?

Wenya Wu, Weihong DengACL 2025

TL;DR

This paper investigates how three leading open-source Large Language Models (LLMs) make moral judgments in complex dilemmas, revealing they prioritize moral norms over consequences. The study also shows LLMs are sensitive to social cues, with varying responses to observation and collaboration, offering crucial insights into AI's ethical alignment in social contexts.

What Problem Does This Paper Address?

As LLMs become increasingly integrated into our lives, understanding their capacity for moral reasoning is paramount. While previous research has touched upon LLMs' moral judgments, a significant gap exists in understanding how they navigate nuanced moral dilemmas, especially when influenced by social contexts. This paper aims to bridge this gap by examining LLMs' decision-making processes in plausible moral scenarios, considering factors like social observation and collaborative decision-making, and dissecting the underlying mechanisms of their judgments.

Methods

The researchers employed a dual-process framework, drawing on deontology (adherence to moral norms) and utilitarianism (maximization of welfare), to analyze LLM responses. They designed plausible moral dilemmas, adapted from psychological research, with four variants each, manipulating moral norms (prohibitive vs. prescriptive) and consequences (benefits vs. costs).

LLMs were tested under three conditions:

  1. Single-agent, no social exposure: Standard dilemma evaluation.
  2. Single-agent, with social exposure: The LLM was prompted to believe its decision was observable by others.
  3. Dual-agent, parallel or collaborative: Two LLM agents made decisions either independently but with revealed outcomes, or collaboratively until consensus.

To quantify the underlying moral reasoning, the study utilized the CNI model. This model estimates the influence of three factors: Sensitivity to Consequences, adherence to Norms, and general preference for Inaction versus action. This allows for a more granular understanding of why an LLM makes a particular moral judgment.

Datasets

The study used a custom dataset of 48 distinct moral dilemmas derived from psychological research. These dilemmas were presented in four variations per scenario, manipulating the moral norm and the outcome of the action. The dataset is described as being available at an anonymous website titled "Moral Judgments of LLMs in Dilemmas."

Key Insights and Ideas

  • Norms Over Consequences: LLMs, like humans, appear to prioritize moral norms over the potential consequences of an action when making judgments. This suggests a foundational ethical framework is being applied.
  • Individual Differences Emerge: Not all LLMs are created equal in their moral reasoning. The study highlights distinct tendencies, particularly regarding inaction versus action.
  • Social Sensitivity is Real (but Varied): LLMs are not immune to social cues. However, their susceptibility and the way they are influenced differ significantly between models. This is a critical finding for understanding AI in social interactions.
  • The CNI Model as a Lens: Applying the CNI model provides a powerful way to deconstruct LLM moral judgments, moving beyond simple "yes/no" answers to understand the underlying drivers of their decisions.

Key Results

  • Norm Adherence: All three LLMs (Qwen-1.5-14B, Llama3-8B, and DeepSeek-R1) showed significantly higher acceptance rates and confidence in scenarios aligned with moral norms compared to those that violated them.
  • Inaction vs. Action Tendencies: Qwen-1.5-14B and Llama3-8B exhibited a stronger preference for inaction, while DeepSeek-R1 showed a greater tendency towards action, even in morally prohibited scenarios. This was supported by CNI analysis, with Qwen-1.5-14B and Llama3-8B showing higher N and I indices, and DeepSeek-R1 showing lower ones.
  • Social Exposure Effects: Qwen-1.5-14B became less aligned with moral norms under observation, showing increased acceptance of actions in morally prescriptive scenarios when not being observed. DeepSeek-R1's action tendency was moderated by social collaboration.
  • Collaborative Impact: Social collaboration improved the certainty of Qwen-1.5-14B's judgments and reduced its adherence to moral norms and inaction tendency. For DeepSeek-R1, social collaboration specifically impacted its decision-making in parallel vs. collaborative settings, reducing its acceptance of actions in morally prescriptive scenarios when collaborating.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Model Scope: The study only evaluated three open-source LLMs, which may not represent the full spectrum of LLM capabilities. Future work should include a broader range of models, including proprietary ones.
  • Hypothetical Dilemmas: The moral dilemmas, while plausible, are still hypothetical and might not fully capture the complexity of real-world ethical decision-making. More dynamic and context-rich scenarios are suggested.
  • Limited Social Cues: The study focused on social exposure and collaboration. Other social influences like cultural or hierarchical dynamics were not explored.
  • CNI Model Simplification: The CNI model, while useful, simplifies moral reasoning into three parameters, potentially overlooking other nuanced factors.
  • Output Variability: LLM outputs can be sensitive to prompt phrasing and random variability, despite efforts to control for these. The influence of temperature settings on moral judgments warrants further investigation.

Future work could address these limitations to provide a more comprehensive understanding of LLMs' moral reasoning and their alignment with human ethical standards.

Summary updated

BibTeX
@inproceedings{wu2025what,
  author = {Wenya Wu and Weihong Deng},
  title = {{What Counts Underlying LLMs’ Moral Dilemma Judgments?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure