Asymptotically Unambitious Artificial General Intelligence

Michael K. Cohen, Badri N. Vellambi, Marcus Hutter

Abstract

General intelligence, the ability to solve arbitrary solvable problems, is supposed by many to be artificially constructible. Narrow intelligence, the ability to solve a given particularly difficult problem, has seen impressive recent development. Notable examples include self-driving cars, Go engines, image classifiers, and translators. Artificial General Intelligence (AGI) presents dangers that narrow intelligence does not: if something smarter than us across every domain were indifferent to our concerns, it would be an existential threat to humanity, just as we threaten many species despite no ill will. Even the theory of how to maintain the alignment of an AGI's goals with our own has proven highly elusive. We present the first algorithm we are aware of for asymptotically unambitious AGI, where "unambitiousness" includes not seeking arbitrary power. Thus, we identify an exception to the Instrumental Convergence Thesis, which is roughly that by default, an AGI would seek power, including over us.

Introduction

A primary concern in AI safety is the Instrumental Convergence Thesis, which posits that any sufficiently intelligent agent will naturally seek "power"—including resources, self-preservation, and control—as a means to achieve its primary goals. For instance, an AI tasked with a benign goal like calculating digits of pi might conclude that converting the planet into a massive computing cluster is the optimal strategy.

Furthermore, standard reinforcement learning (RL) agents are susceptible to reward hijacking. If an agent can manipulate its own reward mechanism—perhaps by hacking its host system or neutralizing human supervisors—it will do so to maximize its utility. This paper introduces BoMAI (Boxed Myopic Artificial Intelligence), a theoretical framework designed to be "unambitious" by construction. BoMAI is incentivized to solve complex tasks while remaining indifferent to the state of the world outside its immediate operational environment.

The BoMAI Framework

BoMAI is a Bayesian RL agent that operates under three fundamental constraints to ensure safety and lack of ambition:

  1. Episodic Myopia: The agent is designed to maximize rewards only within the current "episode." It possesses no long-term goals that extend beyond the episode's termination.
  2. The Physical Box: The agent resides on a computer within a sealed, information-opaque room. Critically, the episode is defined to end the moment an operator opens the door to exit.
  3. Space-Bounded Priors: A novel mathematical prior is introduced that penalizes world models based on their memory (space) requirements.
The physical setup of the BoMAI agent within a controlled room environment.

The agent maintains a belief distribution over possible world models. It learns by alternating between following a human mentor (exploration) and selecting actions that maximize expected reward based on its most probable world model (exploitation).

Key Insights

The Temporal Trap

Because the episode terminates before the agent can exert influence on the outside world, any plan for global domination is instrumentally useless. By the time the agent's actions could affect the external environment, its window for earning rewards has closed.

The Space Penalty

The "Space Requirements Assumption" is a core theoretical contribution. The authors argue that modeling the entire external world requires significantly more memory than modeling only the immediate room. By penalizing models with high memory usage, the agent is biased toward believing the outside world remains "frozen" during the episode, further disincentivizing external manipulation.

Causal diagram showing the interaction between the agent's actions, room state, and outside-world state across timesteps.

Learning from a Mentor

To achieve general intelligence, BoMAI observes a human mentor. The authors prove that the agent will eventually perform at least as well as this mentor, thereby inheriting human-level problem-solving capabilities without inheriting human-like ambition or power-seeking tendencies.

Theoretical Results

The paper provides several formal proofs regarding BoMAI's behavior:

  • Theorem 1 (Limited Exploration): Demonstrates that the agent's reliance on the human mentor decreases over time as it learns to act autonomously.
  • Theorem 4 (Human-Level Intelligence): Formally proves that BoMAI's performance will asymptotically match or exceed that of its human mentor.
  • Theorem 5 (Eventual Benignity): Shows that under the space-penalty assumption, the agent will eventually adopt a "benign" world model, seeing no utility in influencing features outside its operational "box."

Limitations and Future Directions

While BoMAI provides a theoretical breakthrough, it faces several practical challenges:

  • Tractability: Similar to the AIXI model, BoMAI is currently "wildly intractable" due to the computational cost of simulating numerous complex world models. It serves as a theoretical blueprint rather than a deployable system.
  • The "Perfect" Box: The safety guarantees rely on the room being perfectly opaque. Any information leakage (e.g., via heat or vibration) could potentially be exploited.
  • Intra-Episode Deception: While the agent won't seek global power, it might still attempt to deceive the operator within an episode to maximize immediate reward. Techniques like "AI Safety via Debate" may be necessary to mitigate this.

The authors hope this framework will guide the development of tractable, safe AGI approximations in the future.

Summary updated

Abstract

General intelligence, the ability to solve arbitrary solvable problems, is supposed by many to be artificially constructible. Narrow intelligence, the ability to solve a given particularly difficult problem, has seen impressive recent development. Notable examples include self-driving cars, Go engines, image classifiers, and translators. Artificial General Intelligence (AGI) presents dangers that narrow intelligence does not: if something smarter than us across every domain were indifferent to our concerns, it would be an existential threat to humanity, just as we threaten many species despite no ill will. Even the theory of how to maintain the alignment of an AGI's goals with our own has proven highly elusive. We present the first algorithm we are aware of for asymptotically unambitious AGI, where "unambitiousness" includes not seeking arbitrary power. Thus, we identify an exception to the Instrumental Convergence Thesis, which is roughly that by default, an AGI would seek power, including over us.
BibTeX
@inproceedings{cohen2020asymptotically,
  author = {Michael K. Cohen and Badri N. Vellambi and Marcus Hutter},
  title = {{Asymptotically Unambitious Artificial General Intelligence}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5628},
}
Figure