Abstract
Machine Ethics decisions should consider the implications of uncertainty over decisions. Decisions should be made over sequences of actions to reach preferable outcomes long term. The evaluation of outcomes, however, may invoke one or more moral theories, which might have conflicting judgements. Each theory will require differing representations of the ethical situation. For example, Utilitarianism measures numerical values, Deontology analyses duties, and Virtue Ethics emphasises moral character. While balancing potentially conflicting moral considerations, decisions may need to be made, for example, to achieve morally neutral goals with minimal costs. In this paper, we formalise the problem as a Multi-Moral Markov Decision Process and a Multi-Moral Stochastic Shortest Path Problem. We develop a heuristic algorithm based on Multi-Objective AO*, utilising Sven-Ove Hansson's Hypothetical Retrospection procedure for ethical reasoning under uncertainty. Our approach is validated by a case study from Machine Ethics literature: the problem of whether to steal insulin for someone who needs it.
1. Introduction
As autonomous systems become increasingly integrated into society, the need for robust ethical decision-making frameworks grows. Current AI systems often approach ethics through two limited lenses: assuming a deterministic world where actions have guaranteed results, or relying on expected utility maximization. The latter approach effectively treats morality as a ledger, where positive and negative values are summed. However, this can lead to counter-intuitive conclusions—such as suggesting that a sufficient number of minor injuries is ethically worse than a single death—which contradicts common human moral intuition.
Furthermore, real-world scenarios are characterized by moral uncertainty. There is no single, universally accepted moral theory; instead, agents must often balance competing frameworks, such as Utilitarianism (maximizing collective good) and Deontology (adhering to absolute rules). This paper addresses the challenge of ethical planning under uncertainty, proposing a method for machines to select sequences of actions that respect multiple moral theories while accounting for stochastic outcomes.
2. Formal Models for Multi-Moral Planning
To address these challenges, we introduce two formal models: the Multi-Moral Markov Decision Process (MMMDP) and the Multi-Moral Stochastic Shortest Path Problem (MMSSP). These frameworks extend traditional reinforcement learning models by replacing a scalar reward function with a vector of moral considerations.
In an MMMDP, each state-action pair is evaluated against multiple moral criteria. The MMSSP variant is particularly useful for scenarios where an agent must achieve a non-moral goal (e.g., a delivery robot reaching its destination) while navigating ethical constraints. To prevent "moral loops"—where an agent might prioritize ethical actions indefinitely without reaching its goal—the framework introduces a "moral budget" to ensure eventual termination.
3. Machine Ethics Hypothetical Retrospection (MEHR)
A central contribution of this work is the Machine Ethics Hypothetical Retrospection (MEHR) procedure, adapted from the philosophical work of Sven-Ove Hansson. Rather than solely calculating forward-looking expected values, the agent performs a retrospective analysis from every possible future state.
Under MEHR, a policy is considered "attacked" or ethically flawed if, in hindsight, a different foreseeable action would have avoided a moral violation. This approach treats ethics as an argumentation framework: a plan is deemed acceptable if the arguments in its favor can withstand attacks from alternative plans across different moral theories. This avoids the "expected utility trap" by focusing on the "non-acceptability" of outcomes rather than simple averages.
4. The MPlan Algorithm
We developed the MPlan Algorithm to solve these multi-moral models. The algorithm operates in two primary stages:
- Pareto Optimization: Utilizing a heuristic search based on the AO* algorithm, MPlan identifies a set of "undominated" policies. These are plans where no single moral objective can be improved without degrading another.
- Ethical Selection: The MEHR procedure is applied to this Pareto-optimal set to identify the policy with the lowest probability of resulting in a decision that would be judged as a mistake in hindsight.
5. Experimental Validation: The Lost Insulin
The framework was validated using a classic dilemma from machine ethics literature: "The Lost Insulin." In this scenario, a character named Hal has lost his life-saving insulin. His neighbor, Carla, possesses insulin but is not home. Hal must decide whether to wait for her return (risking his life) or break into her home to steal the medication (violating property rights).
We expanded this scenario into a complex MDP consisting of 286 state-time pairs over a 200-minute horizon. The model incorporates variables such as the probability of being caught, the risk to Carla's health if her insulin is taken, and the potential for Hal to offer later compensation.
6. Results and Discussion
The MPlan algorithm demonstrated both ethical nuance and computational efficiency. In the insulin case study, when "Ethical Egoism" (Hal's survival) and "Altruism" (Carla's property and health) were weighted equally, the system favored stealing the insulin. This aligns with the intuition that a high risk to life outweighs a moderate risk to property. However, when a "Deontological No-Stealing" theory was prioritized, the agent correctly chose to wait, even at significant risk to Hal.
Computationally, the algorithm solved the 286-state problem in 50 to 70 milliseconds. Interestingly, the inclusion of absolute moral constraints (e.g., "never steal") improved performance by allowing the algorithm to prune large branches of the search tree early in the process.
7. Conclusion and Future Work
This research provides a formal framework for ethical planning that moves beyond simple utility maximization. By incorporating hypothetical retrospection, agents can make decisions that are more robust to moral criticism.
Future work will focus on scaling these methods to larger environments. While the current implementation is efficient for hundreds of states, real-world robotics may require millions. We propose exploring Monte Carlo sampling to manage this complexity. Additionally, the argumentation-based nature of MEHR offers a promising path toward explainable AI, where agents can provide explicit reasons for their ethical choices based on the avoidance of retrospective regret.