Norm Conflict Resolution in Stochastic Domains

Daniel Kasenberg, Matthias ScheutzAAAI 2018 · DOI

Abstract

Artificial agents operating in social environments must navigate complex, often contradictory norms while accounting for the inherent uncertainty of the real world. This paper introduces a hybrid framework that integrates Linear Temporal Logic (LTL) with Markov Decision Processes (MDPs). By utilizing Conflict Resolution Deterministic Rabin Automata (CRDRA), the system allows agents to "suspend" lower-priority norms at a calculated cost. This approach avoids the "normative explosion" typical of pure logic systems and provides a transparent, reasoning-based alternative to black-box reinforcement learning.

1. Introduction

As robots transition from industrial cages to social spaces, they must adhere to human social and moral norms. However, these norms frequently conflict. For instance, a cleaning robot might be tasked with maintaining floor cleanliness while also being prohibited from disturbing humans. In a scenario where a human is standing over a spill, the robot faces a dilemma. Traditional AI approaches struggle with such conflicts, either becoming paralyzed by logical contradictions or failing to provide interpretable justifications for their actions.

2. The Limits of Existing Approaches

Current norm-adherence strategies generally fall into two categories:

  1. Logic-Based Systems: These use formal rules (like LTL) to define behavior. While precise, they suffer from "normative explosion"—if two rules conflict, the system may conclude that all actions are simultaneously required and forbidden. Furthermore, they are often ill-equipped to handle the probabilistic (stochastic) nature of real-world environments.
  2. Reinforcement Learning (RL): These systems learn through rewards. While effective under uncertainty, they lack transparency. An RL agent cannot explicitly reason about a norm or explain why it chose to violate one rule to satisfy another.

3. A Hybrid Framework for Norm Resolution

Kasenberg and Scheutz propose a framework that combines the strengths of formal logic and probabilistic modeling to bridge this gap.

3.1 Linear Temporal Logic and CRDRAs

Norms are represented using Linear Temporal Logic (LTL), allowing for human-readable definitions like "Always(Clean)" or "Never(Damaged)." To handle conflicts, the authors introduce the Conflict Resolution Deterministic Rabin Automaton (CRDRA). Unlike standard LTL automata that fail upon a violation, a CRDRA allows for the "suspension" of a norm. Each suspension incurs a "violation cost" based on the norm's assigned weight.

3.2 The Product MDP

The agent's environment is modeled as a Markov Decision Process (MDP), capturing the probabilities of various state transitions. By combining the environment MDP with the CRDRAs of all active norms, the authors create a "Product MDP." The agent then uses value iteration to determine a policy that minimizes the total expected discounted violation cost over time. This allows the agent to prioritize norms dynamically based on their relative importance.

4. Experimental Evaluation: The Vacuum Domain

The framework was tested in a simulated vacuum cleaning environment involving a robot with limited battery life and durability, a mobile human, and various hazards (puddles and broken glass).

4.1 Scenario Analysis

  • The Puddle: When encountering a puddle that could damage its hardware, the robot chose to wait for evaporation. It prioritized its "self-preservation" norm over the "cleanliness" norm, as the cost of damage outweighed the temporary violation of cleaning duties.
  • Broken Glass: In a more critical scenario, the robot encountered broken glass that posed a risk to the human. Despite the glass being damaging to the robot itself, the agent chose to vacuum it immediately. The high weight assigned to "human safety" overrode the robot's own "health" norm.
  • Social Interruption: The agent demonstrated the ability to violate minor social norms (e.g., interrupting a human on a phone call) to prevent a major moral violation (e.g., the human stepping on glass).

5. Discussion and Limitations

The weighted suspension model effectively mimics human prioritization, allowing for "lexicographic" behavior where strict hierarchies are maintained through significantly different weights. However, several challenges remain:

  • Scalability: The state space of the Product MDP grows exponentially with the number of norms, leading to the "curse of dimensionality."
  • Uncertainty in Model Parameters: The current model assumes the robot has perfect knowledge of environmental probabilities, which is rarely the case in practice.
  • Philosophical Implications: The use of a "discount factor" for future violations suggests that a violation in the future is less severe than one in the present—a premise that remains ethically debatable.

6. Conclusion

The proposed framework offers a robust path toward building socially intelligent agents capable of navigating moral uncertainty. By treating norms as prioritizable goals rather than binary constraints, robots can make more nuanced, human-like decisions in complex environments. Future research will focus on learning these norms and weights directly from human interaction and natural language.

Summary updated

BibTeX
@inproceedings{kasenberg2018norm,
  author = {Daniel Kasenberg and Matthias Scheutz},
  title = {{Norm Conflict Resolution in Stochastic Domains}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11295},
}
Figure