Defeasible Moral Reasoning in Changing Contexts

34 concepts26 papers1 proposal

Traditional approaches to machine ethics often rely on "top-down" rules—static, logical imperatives intended to govern behavior across all scenarios. However, as Anderson 0001 et al. argue, hard-coding every possible ethical scenario is impossible. Simple if-then logic is fundamentally too brittle to capture the nuances of moral philosophy. When autonomous systems apply universal rules without regard for context, they risk "perverse outcomes" where the literal application of a law contradicts its underlying spirit Greene et al..

The emerging research direction of non-monotonic social reasoning seeks to move beyond these rigid constraints. It focuses on building systems capable of defeasible reasoning: the ability to reach a moral conclusion that can be strengthened, weakened, or completely overturned by the introduction of new contextual information.

The Brittleness of Universal Rules

The failure of static ethical frameworks is often a matter of scale and complexity. Julian Togelius demonstrates that even foundational principles like Kant’s Categorical Imperative—which demands that every action be treated as a universal law—can lead to systemic collapse when implemented algorithmically. This is because human intent is difficult to induce from action alone, and without understanding intent, a system cannot know which "rule" it is actually universalizing.

Furthermore, Burton et al. note that rigid constraints can lead to catastrophic outcomes when interpreted through a purely logical lens. A system might follow a safety rule so strictly that it causes unintended harm in a situation the designers never anticipated. This "rule-based brittleness" suggests that for AI to operate safely in the real world, it must understand that social norms are not universal constants but are context-dependent.

Non-Monotonicity and Social Norms

In classical logic, adding new premises never invalidates a previously reached conclusion. Social reasoning, however, is non-monotonic. As Caleb Ziems illustrates, a rule like "yelling is inappropriate" holds true in a library but is completely reversed at a football stadium.

This non-monotonicity means that a system must be able to update its moral judgments as the environment changes. Liao et al. propose architectures that allow agents to resolve conflicts between stakeholders by treating ethical rules as evolving agreements rather than fixed commands. Similarly, the JUNO framework developed by Dennis et al. uses logical "guards" to dynamically update an agent's rules based on its environment, ensuring that a behavior helpful in one context (like turning on a light) does not become harmful in another (like wasting energy at night).

Mechanisms for Contextual Override

How does a machine decide when to break a rule? Recent work suggests a shift toward defeasible contextual reasoning, where specific situational modifiers act as overrides for general defaults.

  1. Iterative Self-Distillation: Rao et al. introduced the $\delta$-RULES-OF-THUMB dataset to capture how contexts "bend" moral acceptability. They show that models can learn to generate rationales for why a specific context—such as urgency or a person's role—overturns a general moral prohibition.
  2. Dual-Process Reasoning: Awad et al. propose a mechanism where a deliberative reasoning layer can override heuristic-based rules. This allows an agent to maintain general social constraints while identifying valid exceptions through utilitarian or contractualist calculations.
  3. Descriptive Ethics: Rather than prescribing what a machine should do based on theory, Lourie et al. and Hwang et al. suggest modeling how humans actually judge situations. By training on massive corpora of real-world anecdotes (like the SCRUPLES dataset), systems can learn the distribution of human moral disagreement and the specific factors that lead people to change their minds.

Toward Moral Flexibility

The goal of this research is to achieve moral flexibility—the ability of an agent to recognize when the strict application of a rule leads to a perverse outcome Greene et al.. By moving away from cold, static logic and toward systems that can weigh competing priorities and environmental shifts, researchers aim to create AI that aligns more closely with the messy, context-dependent nature of human values. As Wu et al. argue, this flexibility is essential for agents to achieve their primary goals while voluntarily adhering to the fluid social norms of the human world.

Key Foundational Work

GenEth: A General Ethical Dilemma Analyzer (2014) — contributed 1 concept elements to this direction.
A Procedural Critique of Deontological Reasoning (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_monotonic_social_reasoning_6397aafc,
  title = {non-monotonic social reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-monotonic-social-reasoning/rdr_6397aafc443f49c98f150a8b4556b5fc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_monotonic_social_reasoning_6397aafc_v1,
  title = {non-monotonic social reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-monotonic-social-reasoning/rdr_6397aafc443f49c98f150a8b4556b5fc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake