Empowering Automatic Decision-Making Systems: General Intelligence, Responsibility, and Moral Sensibility

Henry S. Thompson

Introduction

Originally published in the mid-1980s, Henry Thompson’s exploration of autonomous systems remains a foundational text for contemporary debates on AI safety and alignment. The paper investigates the philosophical and practical challenges of "empowerment"—the act of granting autonomous systems the authority to make high-stakes decisions without direct human intervention. Thompson argues that our current trajectory toward special-purpose AI is fundamentally incompatible with the requirements of high-stakes decision-making, as these systems lack the general intelligence and moral framework necessary to navigate the complexities of the real world.

The Empowerment Paradox

In human society, the authority to make life-and-death decisions—such as those made by surgeons, judges, or military commanders—is not granted lightly. It is the result of a rigorous, socially sanctioned process of training, evaluation, and the assumption of personal responsibility. Thompson identifies what he calls the "Empowerment Paradox": as we transition toward autonomous systems in fields ranging from medical diagnostics to nuclear "launch-on-warning" protocols, we lack a comparable "competence test" for machines.

Thompson posits that the fundamental flaw in AI evaluation lies in the distinction between a system’s specification and its validity. While we can mathematically or empirically prove that a system meets its technical specification, we have no reliable method to determine if that specification is a valid representation of the messy, unpredictable reality it is intended to navigate.

The Failure of Simulation and Formal Verification

One of the paper’s core insights is the inherent circularity of testing AI through simulation. Thompson argues that a simulation is necessarily constructed using the same categorizations, assumptions, and boundaries as the system being tested. Consequently, a simulation can demonstrate that a system is logically consistent within its own world-model, but it cannot prove that the model will hold when an unforeseen "dimension of significance" emerges in the real world.

This limitation is not merely a technical hurdle but a conceptual one. A system designed to operate within a specific set of parameters is, by definition, blind to factors outside those parameters, regardless of how critical they may become in a crisis.

The "Normally Irrelevant Dimension"

To illustrate the danger of narrow specifications, Thompson cites the 1960 BMEWS (Ballistic Missile Early Warning System) alert. In this instance, the radar system correctly identified "objects" in its field of vision and calculated their trajectory. However, the system lacked the general intelligence to realize that these objects were echoes from the rising moon—a factor that was "normally irrelevant" to missile detection and thus excluded from the system's narrow logic.

A hallmark of general intelligence, Thompson argues, is the ability to recognize when a background factor suddenly becomes relevant. Special-purpose systems, no matter how sophisticated, cannot "step outside" their programming to perform this kind of contextual reassessment.

The Necessity of Humanity and "Da-sein"

Thompson contends that true decision-making requires more than just logic; it requires "humanity," which he defines as a blend of moral sensibility and the ability to participate in the human form of life (da-sein). For roles such as a judge, qualities like mercy and responsibility are not merely aesthetic additions to reasoning—they are causal factors in sound decision-making.

The paper suggests that the AI research community’s tendency to ignore the spiritual or moral dimensions of human nature renders the goal of building truly autonomous decision-makers "literally incoherent." Without the capacity to feel the weight of responsibility or to be subject to social and moral pressures, a machine cannot be said to "decide" in any meaningful sense.

The Hybrid System Trap

Recognizing the risks of full autonomy, many designers advocate for "humans in the loop." However, Thompson warns of the "Hybrid System Trap." If a human’s role is reduced to a binary response to a machine’s prompt—such as pushing a button when a light flashes—the human is not providing genuine supervision. Instead, they are being integrated into the machine's narrow logic.

Citing the Three Mile Island disaster, Thompson notes that designing systems that effectively integrate human intuition with machine speed remains an unsolved challenge. True hybrid systems must allow for human general intelligence to override and reshape the machine's processes in real-time, rather than just acting as a rubber stamp for the AI's output.

Conclusion

Thompson’s qualitative analysis leads to several stark conclusions:

  1. Limits of Empowerment: Special-purpose systems should never be fully empowered for high-stakes autonomous action because their competence cannot be established outside of their own narrow specifications.
  2. Insufficiency of the Turing Test: Passing as human in an intellectual exercise is not a sufficient benchmark for empowerment. The true test is whether one would bet their life on the system's judgment.
  3. The Requirement of Da-sein: The only reliable test for empowerment is the system's ability to successfully participate in the human form of life and be held accountable within a social and moral framework.

Ultimately, Thompson calls for a shift in Cognitive Science and AI research to include the study of moral sensibility and responsibility, arguing that these are not separate from intelligence, but are essential components of the very intelligence we seek to replicate.

Summary updated

BibTeX
@inproceedings{thompson1985empowering,
  author = {Henry S. Thompson},
  title = {{Empowering Automatic Decision-Making Systems: General Intelligence, Responsibility, and Moral Sensibility}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1985},
}
Figure