Logical Limitations to Machine Ethics with Consequences to Lethal Autonomous Weapons

Matthias Englert, Sandra Siebert, Martin ZieglerarXiv 2014 · arXiv

Introduction

The development of Lethal Autonomous Weapons (LASs) is frequently defended by the "moral superiority" argument: the idea that machines, unlike humans, are immune to rage, fatigue, or bias. By encoding the Geneva Conventions directly into their software, proponents argue we can create a more ethical form of warfare. However, this paper demonstrates that "perfect" machine ethics is not just a design challenge, but a mathematical impossibility. By reducing the Halting Problem to a moral dilemma, the authors prove that an autonomous system can be presented with a clear "right" choice and full information, yet remain provably unable to algorithmically decide which action to take.

The Computability of Ethics

While philosophers have long debated the "Trolley Problem" as a matter of moral intuition, this paper shifts the focus to computability. The authors utilize Gedankenexperiments (thought experiments) to bridge the gap between ethics and recursion theory. They construct a series of increasingly complex scenarios culminating in the "Apex" dilemma. This scenario describes a deterministic situation where a unique "right" action exists, all information (including source code) is disclosed, and all actions are deterministic.

To prove that a machine still cannot solve this, the authors use a reduction from the Halting Problem. They demonstrate that determining whether a specific piece of "malicious" code—such as an instruction to fire on a civilian—will ever be executed is undecidable.

Proposition 10 and the Dead-Code Problem

The mathematical core of the paper is Proposition 10. It formally proves that the "dead-code-in-linear-time" problem is undecidable. This means that no algorithm can be guaranteed to identify instructions that will never run, even when the software is restricted to linear execution time. Consequently, no AI can be guaranteed to always identify malicious intent or unethical outcomes in software for all possible inputs. The authors conclude that any AI based on a Turing-equivalent computing device will necessarily fail to identify the "moral" choice in certain scenarios.

The Exculpation Trap and Accountability

The implications of this undecidability extend beyond computer science into the realm of international law. The authors warn of the "Exculpation Trap": if ethical decisions are algorithmically impossible in certain cases, governments might use "mathematical necessity" as a legal shield. War crimes committed by an AI could be dismissed as "provably unavoidable exceptions," creating a "legal limbo" where non-attributable aggression becomes a tool for state actors. This makes autonomous systems attractive for those seeking to commit the "perfect crime" without clear accountability.

Technical and Regulatory Recommendations

To mitigate these risks, the paper concludes with nine specific technical and legal recommendations for the regulation of LASs, including:

  • Mandatory Formal Methods: Requiring the use of software verification tools (such as Coq or Isabelle) to mathematically prove the properties of autonomous systems.
  • Tamper-Proof Logging: Implementing black-box logging using asymmetric encryption (RSA) to ensure that decision-making processes can be audited without the risk of log manipulation.
  • Combatant Identification: A requirement for LASs to carry a "fixed distinctive sign" and a unique ID, similar to the requirements for human combatants under the Geneva Convention.

Conclusion and Future Work

The authors acknowledge that their proofs focus on "worst-case" scenarios derived from recursion theory. While these failures are mathematically possible, the paper does not quantify how likely they are to occur in real-world combat. Future research is suggested to investigate "average-case" complexity—essentially asking how often a machine can be expected to be right, even if it cannot be perfect. Nonetheless, the fundamental proof remains: the desire for perfectly ethical machines is currently at odds with the logical limitations of the hardware and software that control them.

Summary updated

Abstract

Lethal Autonomous Weapons promise to revolutionize warfare -- and raise a multitude of ethical and legal questions. It has thus been suggested to program values and principles of conduct (such as the Geneva Conventions) into the machines' control, thereby rendering them both physically and morally superior to human combatants. We employ mathematical logic and theoretical computer science to explore fundamental limitations to the moral behaviour of intelligent machines in a series of "Gedankenexperiments": Refining and sharpening variants of the Trolley Problem leads us to construct an (admittedly artificial but) fully deterministic situation where a robot is presented with two choices: one morally clearly preferable over the other -- yet, based on the undecidability of the Halting problem, it provably cannot decide algorithmically which one. Our considerations have surprising implications to the question of responsibility and liability for an autonomous system's actions and lead to specific technical recommendations.
BibTeX
@misc{englert2014logical,
  author = {Matthias Englert and Sandra Siebert and Martin Ziegler},
  title = {{Logical Limitations to Machine Ethics with Consequences to Lethal Autonomous Weapons}},
  year = {2014},
  eprint = {1411.2842},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1411.2842},
}
Figure