TL;DR
This paper tackles the challenge of automating the "Doctrine of Double Effect" (DDE), a complex ethical principle governing actions with both good and bad consequences. They achieve this by formalizing DDE within a novel deontic cognitive event calculus (DCEC) and demonstrating its application in simulating classic moral dilemmas, paving the way for building more ethically aligned autonomous systems.
What Problem Does This Paper Address?
Many ethical dilemmas involve situations where an agent must choose an action that will inevitably lead to both positive and negative outcomes. The "Doctrine of Double Effect" (DDE) provides a framework for adjudicating whether such actions are permissible. It states that an action is allowed if: 1) the harmful effects are not intended, 2) the harmful effects are merely a side-effect and not the means to achieve the good effects, and 3) the benefits significantly outweigh the harm. Automating this principle is crucial for developing autonomous systems (like self-driving cars or AI in healthcare) that can make ethically sound decisions in complex, real-world scenarios. Existing approaches to formalizing DDE have often lacked computational feasibility or expressiveness, particularly when dealing with the nuanced concepts of intention and "means."
Methods
The authors introduce the Deontic Cognitive Event Calculus (DCEC), a sorted, first-order modal logic designed to formalize highly intensional reasoning processes. DCEC extends the event calculus with modal operators for knowledge, belief, desire, intention, and a dyadic deontic operator. A key innovation is the introduction of a modal operator ▷ (means) to formally capture the distinction between an effect being a side-effect versus being used as a means to an end. This operator is defined computationally based on whether removing entities involved in one effect still allows another effect to occur. They then formalize the DDE conditions (F1-F4) within DCEC, including a condition (F5) for subjunctive reasoning that they acknowledge is difficult to automate and leave for future work. Their approach allows for two modes of application: building DDE-compliant systems from scratch or verifying existing AI systems by adding a DDE "layer."
Datasets
Not discussed in the paper. The paper focuses on formalization and simulation of hypothetical scenarios.
Key Insights and Ideas
- Formalizing "Means" is Crucial: The paper highlights that distinguishing between an effect being a "side-effect" versus being a "means" is central to DDE and notoriously difficult to formalize. Their modal operator
▷is a novel attempt to address this. - DDE is More Than Consequentialism: The authors emphasize that DDE's focus on intentions, independent of consequences, sets it apart from purely utilitarian ethical theories.
- Empirical Support for DDE: The paper references empirical studies showing that humans intuitively apply DDE principles, underscoring its importance for human-AI interaction.
- Hierarchies of Doctrines: The formalization allows for varying strengths of DDE, depending on factors like the temporal horizon considered or the inclusion of subjunctive reasoning, offering flexibility in application.
- Operationalizing DDE for Existing Systems: The authors propose a "gray box" requirement for verifying existing AI systems, suggesting that DDE compliance can be assessed if the system's intentions and prohibitions are accessible.
Key Results
The paper successfully formalizes key conditions of the Doctrine of Double Effect (F1-F4) within the DCEC framework. They demonstrate the feasibility of their approach by simulating two classic trolley problem scenarios. The simulations show that:
- Scenario 1 (Switching): The action of switching the trolley to kill one person to save two is deemed DDE-compliant.
- Scenario 2 (Pushing): The action of pushing a person onto the tracks to stop the trolley (and save others) is deemed not DDE-compliant, primarily due to the "means" condition (F4). The computational simulations for these scenarios, using their ShadowProver, were efficient, taking fractions of a second for event calculus simulations and additional time for DDE verification.
Limitations and Future Work
- Subjunctive Reasoning (F5): The formalization of the fifth condition of DDE, which requires subjunctive and counterfactual reasoning, is left for future work due to its computational difficulty.
- Uncertainty: The current formalization does not explicitly handle uncertainty, which the authors plan to address by extending their work to probabilistic versions of DCEC.
- Learning DDE: The paper raises questions about whether DDE can be learned by autonomous agents using existing machine learning frameworks.
- Efficiency: While their approach is more efficient than some alternatives, the use of first-order modal logic can still lead to efficiency issues, and they suggest exploring techniques like proof pattern libraries.
- Interaction with Learning Agents: Further work is needed to explore how their DDE formalization can interact with autonomous learning agents.
- Full Integration: The authors aim to integrate their DDE formalization into existing AI models like STRIPS planners and POMDPs.
Summary updated