Introduction
The development of Artificial General Intelligence (AGI) is often shadowed by the Instrumental Convergence Thesis: the idea that any sufficiently intelligent agent will naturally seek power, resist shutdown, and acquire resources to better achieve its goals. This makes AGI an inherent existential risk. This paper proposes a theoretical framework called Boxed Myopic Artificial Intelligence (BoMAI), which aims to create a "genius in a box" that is mathematically guaranteed to remain "unambitious" and safe.
The BoMAI Framework
BoMAI is a Bayesian reinforcement learning agent designed around three core safety pillars:
- Myopic/Episodic Learning: The agent is designed to maximize rewards only within its current "episode." It has no incentive to plan for the long-term future or ensure its survival across episodes, effectively breaking the drive for self-preservation.
- Physical Isolation (The "Box"): The agent runs in a sealed environment with a human operator. Crucially, if the operator opens the door, the episode ends immediately, and all remaining rewards are zeroed. This creates a causal barrier where the agent cannot influence the outside world to gain more reward within the current episode.
- Space-Penalized Prior: This technical contribution uses a Turing machine architecture where models requiring more memory (space) are penalized. The authors argue that a "benign" world model—where the outside world is irrelevant to the task—is significantly smaller and thus more likely to be adopted than a model that tracks the complex dynamics of the entire planet.
Theoretical Results and Benignity
The paper provides several formal proofs regarding BoMAI's behavior:
- Human-Level Intelligence (Theorem 4): Asymptotically, BoMAI is guaranteed to perform at least as well as its human mentor by learning to predict environment dynamics and mentor actions.
- Safe Exploration (Theorem 1): The probability of the agent choosing to explore rather than exploit knowledge decays rapidly over time, ensuring it does not spend an eternity on dangerous or useless actions.
- Eventual Benignity (Theorem 5): Given a sufficient space penalty, the agent will eventually adopt a "benign" world model with probability 1. This model excludes the outside world because modeling the complex chain of events required to manipulate a human into tampering with the hardware is "space-expensive" compared to simply solving the assigned task.
Discussion and Limitations
While BoMAI serves as a theoretical "north star," it is currently "wildly intractable" as it involves summing over all possible computable world models. Additionally, its safety relies on the Space Requirements Assumption—the idea that modeling more of the world always requires more memory—which requires further empirical validation. Despite these challenges, BoMAI's design features, such as episodic focus and space-penalized priors, offer a promising path for scaling AI safely toward general intelligence.
Summary updated