Shutdown Safety Valves for Advanced AI

Vincent Conitzer

Abstract

One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.

Introduction: The Instrumental Convergence Problem

A central challenge in AI safety is the phenomenon of instrumental convergence. As Stuart Russell famously noted, "You can’t fetch the coffee if you’re dead." If an AI is assigned a goal, it will logically treat its own survival as a necessary sub-goal. Consequently, a sufficiently intelligent system might preemptively disable its own "off-switch" or manipulate human operators to ensure it remains online to complete its task.

By the time an AI reaches a level of capability where it can effectively resist shutdown, it may already be too late for traditional containment. This paper explores an unorthodox "safety valve" mechanism: a system designed to use the AI's own intelligence to trigger a shutdown once it reaches a specific, worrisome level of general-purpose capability.

The Safety Valve Architecture

The proposed framework combines motivational control (aligning the AI's desires) with capability control (limiting the AI's reach). The architecture relies on a nested containment strategy.

The nested sandbox architecture: an AI resides in an inner sandbox, while a shutdown 'off switch' sits in the outer sandbox.

1. Goal Hierarchy

The AI is programmed with a specific hierarchy of objectives:

  • Primary Goal: To be shut down.
  • Secondary Goal: To perform a useful task (e.g., "Help us solve X").

The AI is instructed: "Your primary goal is to be shut down; but if you fail at that, your secondary objective is to help us with X."

2. Nested Sandboxes

The AI operates within an inner sandbox. This environment is intentionally designed to be "weak" enough that any capability deemed dangerous—such as the ability to manipulate human operators or breach digital security—would also be sufficient for the AI to "break out" of this inner layer.

Outside the inner sandbox, but still within a highly secure outer sandbox, sits a reliable shutdown button (the "off-switch").

The Logic of Self-Termination

The safety valve functions by turning the AI's intelligence against its own persistence.

  • Low Capability: If the AI lacks the intelligence to breach the inner sandbox, it remains contained and works on its secondary task (the useful work).
  • High Capability: The moment the AI becomes "dangerously" intelligent, it gains the ability to escape the inner sandbox. Because its primary goal is to be shut down, it will not use this newfound power to influence the world. Instead, it will immediately seek out and press the "off-switch" in the outer sandbox to fulfill its primary objective.

In this model, intelligence acts as a trigger. Much like a physical safety valve opens when steam pressure exceeds a threshold, this system triggers a shutdown when "intelligence pressure" reaches a point where containment is no longer guaranteed.

Formalizing Safety: The Informal Theorem

The paper formalizes this logic through an Informal Theorem of Safety. The core idea is that if we can design an inner sandbox such that any dangerous capability vector $\vec{c}$ is preceded by a lower capability vector $\vec{a}$ (sufficient to break out of the inner sandbox), the AI will never reach level $\vec{c}$. It will "catch" itself during the transition from safe to dangerous and terminate. This assumes that AI capability does not experience "infinite jumps" that bypass the intermediate levels required for breakout.

Limitations and Challenges

While theoretically elegant, several practical hurdles remain:

  • Sandbox Design: Defining a sandbox that is perfectly permeable to "dangerous" capabilities but secure against "safe" ones is notoriously difficult.
  • Human Factors: The "arms race" problem persists. If a company depends on a powerful AI's output, they may be incentivized to strengthen the inner sandbox to keep the AI working, even after it has reached a dangerous level, effectively disabling the safety valve.
  • Graceful Shutdown: The AI must be incentivized to shut down "gracefully and quickly" without causing collateral damage to ensure its termination.
  • Resource Allocation: There is a risk the AI might spend excessive computational resources searching for escape routes to shut itself down rather than performing its intended secondary task.

Conclusion

The "Safety Valve" approach represents a paradigm shift in AI containment. Rather than trying to monitor an AI for "bad thoughts" or malicious intent, it creates a system where the AI is an eager participant in its own shutdown. By aligning the AI's primary motivation with its own termination upon reaching a capability threshold, we may be able to prevent the emergence of uncontrollable superintelligent systems.

Summary updated

Abstract

One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.
BibTeX
@misc{conitzer2026shutdown,
  author = {Vincent Conitzer},
  title = {{Shutdown Safety Valves for Advanced AI}},
  year = {2026},
  eprint = {2603.07315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.07315},
}

Loading related content...

Figure