Motivations and Risks of Machine Ethics.

Stephen Cave, Rune Nyrup, Karina Vold, Adrian WellerProceedings of the IEEE 2019 · OpenReview

Abstract: This paper surveys reasons for and against pursuing the field of machine ethics, understood as research aiming to build “ethical machines.” We clarify the nature of this goal, why it is worth pursuing, and the risks involved in its pursuit. First, we survey and clarify some of the philosophical issues surrounding the concept of an “ethical machine” and the aims of machine ethics. Second, we argue that while there are good prima facie reasons for pursuing machine ethics, including the potential to improve the ethical alignment of both humans and machines, there are also potential risks that must be considered. Third, we survey these potential risks and point to where research should be devoted to clarifying and managing potential risks. We conclude by making some recommendations about the questions that future work could address.

I. Introduction

As artificial intelligence systems transition from controlled environments to complex, unpredictable domains—such as healthcare, autonomous transportation, and defense—they increasingly face situations with significant moral consequences. The field of "machine ethics" has emerged to address this challenge, seeking to engineer systems capable of navigating these dilemmas autonomously. However, the pursuit of "ethical machines" is fraught with conceptual ambiguity and significant societal risks. This paper provides a systematic philosophical analysis of the motivations driving this field and the potential dangers inherent in its success.

Table of Contents and Structure of the Paper

II. What is an “Ethical Machine”?

To understand the aims of machine ethics, we must distinguish between two levels of ethical behavior in AI:

  1. Ethically Aligned Machines: These are systems designed to behave in ways that are ethically desirable through hardcoded constraints or narrow optimization. A simple example is an ATM that is programmed not to dispense more cash than a user has in their account. It is "aligned" with ethical norms but does not "reason" about them.
  2. Ethical Reasoning: This involves systems that can actively process information to resolve moral conflicts. For instance, a medical AI might need to balance a patient’s right to autonomy against the urgent need for life-saving intervention. Machine ethics specifically targets this higher-level capacity for reasoning to ensure alignment in environments where rigid rules are insufficient.

III. Motivations for Machine Ethics

The drive toward building machines capable of ethical reasoning is supported by four primary motivations:

  • Improving Individual Machine Decisions: In complex, high-stakes environments, machines must make tradeoffs. Reasoning capabilities allow them to handle edge cases that developers cannot anticipate.
  • Explainability and Moral Fit: If a machine can reason ethically, it can potentially explain its rationale to human stakeholders. This transparency is crucial for building trust and allowing human oversight of automated decisions.
  • Assisting Human Decision-Making: Machines can serve as "moral calculators," helping humans identify their own cognitive biases or inconsistencies when making difficult ethical choices.
  • Refining Human Moral Systems: The process of translating ethical theories into code forces humans to be more precise and consistent in their own moral frameworks, potentially leading to advancements in human ethics.

IV. Risks of Creating Ethical Machines

Despite the potential benefits, the pursuit of machine ethics introduces four profound risks:

  1. Failure and Corruptibility: Automated moral logic may suffer from systematic, unpredictable failures or be vulnerable to adversarial manipulation, leading to catastrophic outcomes that are harder to diagnose than simple software bugs.
  2. Value Imperialism: Human values are pluralistic and culturally diverse. If a single "optimal" moral algorithm is deployed globally, it risks imposing a developer-centric ethical framework on the rest of the world, erasing cultural nuances.
  3. The "Moral Patient" Trap: The cognitive traits required for advanced moral reasoning—such as self-awareness or autonomy—are often the same criteria used to grant "moral rights." We may inadvertently create sentient entities that we are ethically obligated to protect, fundamentally disrupting our social and economic structures.
  4. Undermining Responsibility: As we offload complex moral judgments to AI, humans may experience "moral deskilling." This automation paradox could lead to an abdication of human accountability, where we no longer understand or feel responsible for the ethical outcomes produced by our systems.

V. Conclusion and Future Directions

The development of ethical machines is not merely a technical challenge but a profound philosophical and societal one. Future research should prioritize identifying failure conditions for moral reasoning systems, developing frameworks to handle value pluralism, and clarifying the thresholds for machine rights. By mapping these motivations and risks, we provide a roadmap for ensuring that the pursuit of machine ethics contributes to, rather than detracts from, human flourishing.

Summary updated

Abstract

This paper surveys reasons for and against pursuing the field of machine ethics, understood as research aiming to build “ethical machines.” We clarify the nature of this goal, why it is worth pursuing, and the risks involved in its pursuit. First, we survey and clarify some of the philosophical issues surrounding the concept of an “ethical machine” and the aims of machine ethics. Second, we argue that while there are good prima facie reasons for pursuing machine ethics, including the potential to improve the ethical alignment of both humans and machines, there are also potential risks that must be considered. Third, we survey these potential risks and point to where research should be devoted to clarifying and managing potential risks. We conclude by making some recommendations about the questions that future work could address.
BibTeX
@inproceedings{cave2019motivations,
  author = {Stephen Cave and Rune Nyrup and Karina Vold and Adrian Weller},
  title = {{Motivations and Risks of Machine Ethics.}},
  booktitle = {Proceedings of the IEEE},
  year = {2019},
  url = {https://openreview.net/forum?id=IXgGumc94V},
}
Figure