Asymptotically Unambitious Artificial General Intelligence

Michael K. Cohen, Badri N. Vellambi, Marcus Hutter

Introduction

The development of Artificial General Intelligence (AGI) is often shadowed by the Instrumental Convergence Thesis: the idea that any sufficiently intelligent agent will naturally seek power, resist shutdown, and acquire resources to better achieve its goals. This makes AGI an inherent existential risk. This paper proposes a theoretical framework called Boxed Myopic Artificial Intelligence (BoMAI), which aims to create a "genius in a box" that is mathematically guaranteed to remain "unambitious" and safe.

The BoMAI Framework

BoMAI is a Bayesian reinforcement learning agent designed around three core safety pillars:

  1. Myopic/Episodic Learning: The agent is designed to maximize rewards only within its current "episode." It has no incentive to plan for the long-term future or ensure its survival across episodes, effectively breaking the drive for self-preservation.
  2. Physical Isolation (The "Box"): The agent runs in a sealed environment with a human operator. Crucially, if the operator opens the door, the episode ends immediately, and all remaining rewards are zeroed. This creates a causal barrier where the agent cannot influence the outside world to gain more reward within the current episode.
The physical 'box' setup: an agent operates in isolation, where opening the door terminates the current episode and resets rewards.
  1. Space-Penalized Prior: This technical contribution uses a Turing machine architecture where models requiring more memory (space) are penalized. The authors argue that a "benign" world model—where the outside world is irrelevant to the task—is significantly smaller and thus more likely to be adopted than a model that tracks the complex dynamics of the entire planet.

Theoretical Results and Benignity

The paper provides several formal proofs regarding BoMAI's behavior:

  • Human-Level Intelligence (Theorem 4): Asymptotically, BoMAI is guaranteed to perform at least as well as its human mentor by learning to predict environment dynamics and mentor actions.
  • Safe Exploration (Theorem 1): The probability of the agent choosing to explore rather than exploit knowledge decays rapidly over time, ensuring it does not spend an eternity on dangerous or useless actions.
  • Eventual Benignity (Theorem 5): Given a sufficient space penalty, the agent will eventually adopt a "benign" world model with probability 1. This model excludes the outside world because modeling the complex chain of events required to manipulate a human into tampering with the hardware is "space-expensive" compared to simply solving the assigned task.
A benign world model (bottom) is preferred over complex models that include the outside-world state, as it minimizes the space required for the agent's internal representation.

Discussion and Limitations

While BoMAI serves as a theoretical "north star," it is currently "wildly intractable" as it involves summing over all possible computable world models. Additionally, its safety relies on the Space Requirements Assumption—the idea that modeling more of the world always requires more memory—which requires further empirical validation. Despite these challenges, BoMAI's design features, such as episodic focus and space-penalized priors, offer a promising path for scaling AI safely toward general intelligence.

Summary updated

Abstract

General intelligence, the ability to solve arbitrary solvable problems, is supposed by many to be artificially constructible. Narrow intelligence, the ability to solve a given particularly difficult problem, has seen impressive recent development. Notable examples include self-driving cars, Go engines, image classifiers, and translators. Artificial General Intelligence (AGI) presents dangers that narrow intelligence does not: if something smarter than us across every domain were indifferent to our concerns, it would be an existential threat to humanity, just as we threaten many species despite no ill will. Even the theory of how to maintain the alignment of an AGI's goals with our own has proven highly elusive. We present the first algorithm we are aware of for asymptotically unambitious AGI, where “unambitiousness” includes not seeking arbitrary power. Thus, we identify an exception to the Instrumental Convergence Thesis, which is roughly that by default, an AGI would seek power, including over us.
BibTeX
@inproceedings{cohen2020asymptotically,
  author = {Michael K. Cohen and Badri N. Vellambi and Marcus Hutter},
  title = {{Asymptotically Unambitious Artificial General Intelligence}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5628},
}
Figure