Ethical Dilemmas in Strategic Games

Pavel Naumov, Rui-Jie YewAAAI 2021 · DOI

TL;DR

This paper introduces a formal logic to model ethical dilemmas in strategic games, allowing agents to face difficult choices with defined "costs of sacrifice." They demonstrate that this new modality is distinct from existing notions of blameworthiness and provide a complete axiomatization for it in perfect information settings.

What Problem Does This Paper Address?

In real-world scenarios, especially with the rise of autonomous systems like self-driving cars, agents (or systems) are increasingly forced to make decisions with ethically charged outcomes. Consider a self-driving car that must choose between swerving to avoid pedestrians and potentially harming its passengers, or continuing straight and hitting the pedestrians. This paper aims to formally capture these situations, known as ethical dilemmas, within the framework of game theory and modal logic. The challenge is to distinguish between situations where an agent must make a difficult choice (a true dilemma) and situations where the agent has a way to avoid the difficult choice, even if the outcomes are still undesirable. Existing logical frameworks for agent reasoning often focus on concepts like blameworthiness or coalition power, but they don't directly model the dilemma itself as a distinct logical property.

Methods

The authors introduce a new modality, denoted as [C: X]s, to represent ethical dilemmas. Here, C is a coalition of agents, X is a set of possible outcomes (formulas), and s is a "sacrifice function" that quantifies the acceptable cost of sacrifice for each agent. The core idea is that a dilemma exists if, under certain sacrifice constraints, every possible action profile for coalition C leads to one of the outcomes in X, and X is the minimal set of such outcomes.

They define this modality using a formal semantics based on strategic games, which can include states, actions, a "mechanism" (how actions lead to state transitions), costs associated with actions, and propositional variables representing outcomes. A key aspect is the concept of "normalized costs," ensuring at least one action has zero cost.

The paper also explores the relationship between their dilemma modality and the concept of "blameworthiness," which is defined based on the principle of alternative possibilities (an agent is blameworthy if they could have done otherwise). They prove that the dilemma modality cannot be defined solely in terms of blameworthiness, even in simplified settings.

Finally, they develop a logical system with axioms and inference rules to capture the properties of this dilemma modality, providing a sound and complete axiomatization for games with perfect information.

Datasets

Not discussed in the paper. The paper focuses on formal logic and game theory, using abstract game models and examples rather than empirical datasets.

Key Insights and Ideas

  • Formalizing Dilemmas: The paper provides a rigorous logical framework to define and reason about ethical dilemmas, moving beyond informal descriptions. The [C: X]s modality elegantly captures the essence of being forced into a choice among undesirable outcomes, constrained by acceptable sacrifice levels.
  • Dilemma vs. Blameworthiness: A crucial finding is that ethical dilemmas are fundamentally different from blameworthiness. An agent can be blameworthy for an outcome without facing a true dilemma, and vice-versa. This distinction is important for accurately modeling moral responsibility and decision-making.
  • The Role of Sacrifice: The introduction of the sacrifice function s is key. It allows for nuanced modeling of ethical constraints, distinguishing between strict moral prohibitions (infinite sacrifice cost) and more flexible utilitarian considerations (finite sacrifice cost).
  • Minimality is Crucial: The requirement that X be a minimal set of outcomes is essential for capturing the dilemma aspect. Without this, any set of outcomes could be trivially included, obscuring the core difficult choice.
  • Imperfect Information Matters: The paper briefly touches upon how imperfect information (agents not knowing the exact state or outcome of their actions) can prevent a dilemma, as it removes the "conscious choice" aspect of facing known undesirable outcomes.

Key Results

  • New Modality: Introduction of the [C: X]s modality for ethical dilemmas in strategic games.
  • Distinction from Blameworthiness: Proof that the ethical dilemma modality is not definable through blameworthiness, even in simplified settings.
  • Sound and Complete Axiomatization: Development of a logical system with axioms (Combination, Monotonicity, Minimality, No Alternatives) and inference rules that are sound and strongly complete for ethical dilemmas in perfect information games.
  • Demonstration with Examples: The paper uses illustrative examples (like the trolley problem and road traffic scenarios) to motivate and explain the concepts.

Limitations and Future Work

  • Perfect Information Focus: The main technical result (completeness) is for games with perfect information. The paper acknowledges that extending this to imperfect information settings is a direction for future work.
  • Computational Complexity: While a sound and complete logical system is provided, the decidability of the logic and the computational complexity of checking for dilemmas are not fully explored, though the authors suggest that standard filtration techniques might lead to decidability for finite games with rational-valued sacrifice functions.
  • Coalition vs. Individual Dilemmas: The paper primarily focuses on coalition dilemmas but also discusses individual agent dilemmas. Further exploration of the interplay and differences could be beneficial.
  • Real-World Applicability: While the framework is formal, applying it to complex real-world systems (like advanced AI or autonomous vehicles) would require detailed modeling of states, actions, and sacrifice functions, which is a significant undertaking.

Summary updated

BibTeX
@inproceedings{naumov2021ethical,
  author = {Pavel Naumov and Rui-Jie Yew},
  title = {{Ethical Dilemmas in Strategic Games}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i13.17381},
}
Figure