Moral Planning Agents with LTL Values

Umberto Grandi, Emiliano Lorini, Timothy ParkerIJCAI 2023 · DOI

TL;DR

This paper introduces a formal framework for "Moral Planning Agents" (MPAs) that can evaluate and compare potential action plans based on a set of prioritized ethical values expressed in Linear Temporal Logic (LTL). The authors propose novel evaluation criteria, including optimistic, pessimistic, and responsibility-based approaches, to help agents make ethically sound decisions in complex, multi-agent scenarios.

What Problem Does This Paper Address?

In the realm of Artificial Intelligence, particularly in multi-agent systems and robotics, agents often need to make decisions that go beyond mere task completion. They must consider ethical implications and values, especially when their actions can affect others or have broader societal consequences. However, formalizing and computationally reasoning about these ethical considerations in a planning context is challenging. Existing approaches often struggle with conflicting values, the prioritization of these values, and how to evaluate plans when agents have incomplete information about each other's intentions. This paper addresses the need for a robust framework that allows agents to compare and select plans based on a hierarchy of ethical values, moving beyond simple goal achievement to encompass moral reasoning.

Methods

The core of the paper lies in its formalization of Moral Planning Agents (MPAs) and their decision-making process.

  • Value Representation: Ethical values are represented using formulas of Linear Temporal Logic (LTLf), which allows for expressing properties over sequences of states and actions. These values are organized into a prioritized value base, where higher-priority values take precedence over lower-priority ones.
  • Plan Evaluation: The paper defines two main types of plans: joint plans (executed by a coalition of agents) and individual plans (executed by a single agent).
  • Joint Plan Evaluation: Joint plans are evaluated by first generating the history (sequence of states and actions) that results from executing the plan. This history is then compared against the value base.
  • Individual Plan Evaluation: For individual plans, where an agent might not know the actions of others, the paper introduces several evaluation criteria:
  • Optimistic (Risk-Seeking): Compares plans based on the best possible outcome an agent could achieve, assuming other agents' actions are favorable.
  • Pessimistic (Risk-Averse): Compares plans based on the worst possible outcome, assuming other agents' actions are unfavorable.
  • Responsibility-Based: This is a novel contribution, introducing notions of "anticipated passive blameworthiness" (failing to prevent a value violation) and "anticipated active blameworthiness" (causing a value violation). These criteria consider not only whether values are satisfied but also whether the agent can be held responsible for their violation.
  • Formal Framework: The model is grounded in situation calculus and uses a multi-agent action theory to define action effects and preconditions.

Datasets

Not discussed in the paper. The paper focuses on a formal framework and uses illustrative examples rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Prioritized Ethical Values are Crucial: Simply having a set of values isn't enough; their prioritization is key to resolving conflicts and making meaningful decisions, especially in complex scenarios. The LTLf representation allows for nuanced expression of these values.
  • Optimistic vs. Pessimistic is Not Always Enough: In many realistic scenarios, purely optimistic or pessimistic evaluation can lead to indecisiveness. This highlights the need for more sophisticated reasoning, particularly when dealing with potential negative consequences.
  • Responsibility is a Key Differentiator: Introducing concepts of active and passive blameworthiness provides a more nuanced way to evaluate plans. An agent might choose a plan that is not strictly "optimal" in a pessimistic sense but avoids being blameworthy, which is a more ethically robust choice.
  • LTLf's Expressiveness is Powerful: Using LTLf for values allows for capturing complex temporal properties of actions and states, which is essential for reasoning about ethical behavior over time.

Key Results

  • PSPACE-Completeness: The paper establishes that the problems of optimistic and pessimistic comparison of individual plans, as well as determining an agent's blameworthiness, are PSPACE-complete. This indicates that while these problems are computationally challenging, they are within the bounds of what can be solved by current AI systems.
  • Resilient and Fragile Domains: The authors identify specific types of MPAP domains ("k-resilient" and "k-fragile") where optimistic and pessimistic comparisons, respectively, become indecisive, further motivating the need for responsibility-based evaluation.
  • Illustrative Examples: The paper uses a "toy-sharing" scenario to demonstrate how the different evaluation criteria (optimistic, pessimistic, responsibility) can lead to different plan choices and how these choices align with ethical considerations.

Limitations and Future Work

  • Shared Value Base: The current model assumes all agents share the same value base. Future work could explore scenarios with individual or differing value bases.
  • Perfect Information and Determinism: The model assumes perfect information and deterministic actions. Relaxing these assumptions to handle uncertainty and partial observability is a key future direction.
  • Incomparability: While the paper acknowledges that qualitative comparison can lead to incomparable plans (which can be useful for ethical dilemmas), future work could explore methods for resolving such incomparabilities.
  • Epistemic Operators: Extending LTLf to include epistemic operators would allow for reasoning about agents' beliefs and knowledge, which is crucial for more advanced AI systems like LLMs.
  • Indirect Responsibility: Exploring indirect notions of responsibility, beyond direct causal contribution, is another avenue for future research.
  • Real-World Implementations: The authors aim to develop real-world implementations of their model to demonstrate its capabilities.

Summary updated

BibTeX
@inproceedings{grandi2023moral,
  author = {Umberto Grandi and Emiliano Lorini and Timothy Parker},
  title = {{Moral Planning Agents with LTL Values}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/47},
}
Figure