The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation

Beishui Liao, Pere Pardo, Marija Slavkovik, Leendert van der Torre

TL;DR

This paper introduces "Jiminy," an AI advisor that helps autonomous systems make ethical decisions by synthesizing the moral viewpoints of various stakeholders. It uses formal argumentation to resolve conflicts and reach agreements, offering a principled way to integrate diverse ethical considerations into AI behavior.

What Problem Does This Paper Address?

As autonomous systems become more integrated into society, a critical challenge arises: how do we ensure their ethical behavior aligns with the diverse moral expectations of all affected parties? Simply programming a single ethical framework into an AI can lead to biased or unacceptable outcomes. The authors highlight the need for a mechanism that can reconcile potentially conflicting moral instructions from different stakeholders, such as users, manufacturers, and regulators, to arrive at a morally sound decision. They aim to move beyond simple rule-following to a more nuanced approach that can handle moral dilemmas and provide explanations for its ethical reasoning.

Methods

The core of the Jiminy advisor is built upon two key techniques:

  • Normative Systems: Each stakeholder's ethical views are modeled as a "normative system." These systems define what is considered desirable or undesirable behavior in various situations. The paper discusses both table-based and logic-based representations for these systems, allowing for the formalization of regulative, constitutive, and permissive norms.
  • Formal Argumentation: To resolve conflicts and reach agreements among these normative systems, Jiminy employs formal argumentation. This involves constructing arguments based on the norms and facts, and then analyzing the relationships between these arguments (attacks and defeats). The system uses a multi-level approach to check for and resolve moral dilemmas:
  • Individual Frameworks: Analyzing each stakeholder's framework independently.
  • Combined Framework: Merging all stakeholder arguments into a single framework.
  • Integrated Framework: Combining all normative systems into a unified theory.
  • Jiminy Framework: Incorporating stakeholder priority rules to resolve remaining conflicts.

This structured approach allows Jiminy to progressively refine its recommendations and identify the most robust ethical consensus.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Stakeholder-Centric Ethics: The paper strongly advocates for a stakeholder-centric approach to AI ethics, arguing that no single group should unilaterally define an AI's morality.
  • Argumentation as a Reconciliation Tool: Formal argumentation provides a powerful framework for not just identifying ethical conflicts but also for systematically resolving them by analyzing the structure of arguments and their relationships.
  • Multi-Level Dilemma Resolution: The proposed four-level approach to dilemma checking is crucial. It allows for a gradual integration of stakeholder views, starting with independent analysis and moving towards a unified decision, ensuring that simpler resolutions are preferred.
  • Explainability is Intrinsic: The argumentation-based approach naturally supports explainability. The system can trace the reasoning process, identify why certain arguments were accepted or rejected, and present this as a justification for its ethical recommendations.
  • Norms as Arguments: Framing norms as rules that generate arguments is a key insight, allowing for a unified treatment of ethical reasoning within an argumentation framework.

Key Results

The paper presents a formal framework for building an ethical advisor that can handle moral dilemmas arising from multiple stakeholders. While specific quantitative results are not the focus, the paper demonstrates through detailed examples (like the smart speaker dilemma) how the Jiminy advisor can:

  • Model diverse stakeholder norms.
  • Identify moral dilemmas arising from conflicting obligations.
  • Resolve these dilemmas through a hierarchical argumentation process.
  • Provide a moral recommendation to an autonomous agent.
  • Offer explanations for its decisions by referencing the argumentation process.

The framework is shown to be capable of progressively resolving conflicts across four levels, with the ultimate goal of producing a morally agreeable outcome.

Limitations and Future Work

  • Scalability: The paper acknowledges that the argumentation approach can lead to an exponential increase in complexity with more stakeholders and norms, suggesting the need for efficient algorithms, especially for real-time applications.
  • Stakeholder Prioritization: While the Jiminy framework includes mechanisms for stakeholder selection norms, the paper notes that determining these priorities can be domain-specific and may require further research.
  • "Off" Switch: The authors mention the possibility of users having an "off" switch for the Jiminy, which would effectively disable its ethical constraints. The implications and design of such features are left for future work.
  • Learning Moral Theories: The paper focuses on integrating pre-defined stakeholder norms. Future work could explore how agents might learn or adapt their own moral theories.
  • Domain-Specific Theories: While the framework is abstract, adapting it to specific domains (like law) and incorporating domain-specific reasoning techniques is a promising avenue.

Summary updated

Abstract

An autonomous system is constructed by a manufacturer, operates in a society subject to norms and laws, and interacts with end users. All of these actors are stakeholders affected by the behavior of the autonomous system. We address the challenge of how the ethical views of such stakeholders can be integrated in the behavior of an autonomous system. We propose an ethical recommendation component called Jiminy which uses techniques from normative systems and formal argumentation to reach moral agreements among stakeholders. A Jiminy represents the ethical views of each stakeholder by using normative systems, and has three ways of resolving moral dilemmas that involve the opinions of the stakeholders. First, the Jiminy considers how the arguments of the stakeholders relate to one another, which may already resolve the dilemma. Secondly, the Jiminy combines the normative systems of the stakeholders such that the combined expertise of the stakeholders may resolve the dilemma. Thirdly, and only if these two other methods have failed, the Jiminy uses context-sensitive rules to decide which of the stakeholders take preference over the others. At the abstract level, these three methods are characterized by adding arguments, adding attacks between arguments, and revising attacks between arguments. We show how a Jiminy can be used not only for ethical reasoning and collaborative decision-making, but also to provide explanations about ethical behavior.
BibTeX
@article{liao2023jiminy,
  author = {Beishui Liao and Pere Pardo and Marija Slavkovik and Leendert van der Torre},
  title = {{The Jiminy Advisor: Moral Agreements among Stakeholders Based on Norms and Argumentation}},
  journal = {J. Artif. Intell. Res.},
  year = {2023},
  doi = {10.1613/jair.1.14368},
}
Figure