A Definition of Catastrophic Performance

Henrik Marklund, Alex Infanger, Benjamin Van Roy

TL;DR

AI alignment research often struggles to distinguish between "merely suboptimal" outcomes and truly "catastrophic" ones. This paper introduces a formal definition of catastrophic performance by comparing AI behavior to two new reference points: a baseline value (what happens if we do nothing) and an orthogonal value (the value of a trajectory completely unrelated to human preferences).

What Problem Does This Paper Address?

In the field of reinforcement learning (RL), performance is typically measured relative to the optimal policy (often called "regret"). While useful for optimization, this metric fails to capture the severity of failure. An agent might be "suboptimal" but still highly beneficial, or it might be "suboptimal" and cause a global disaster. Standard theory can't tell the difference.

Furthermore, existing "worst-case" metrics often assume a malicious agent—one that actively tries to minimize human reward. However, the primary concern in AI safety isn't necessarily a "terminator" scenario where the AI hates us; it's a "sorcerer's apprentice" scenario where a highly capable but misaligned AI pursues a goal that accidentally destroys everything we care about as a byproduct. We need a formal way to quantify this "accidental catastrophe."

Methods

The authors propose a framework based on Markov Decision Processes (MDPs) and Algorithmic Information Theory to define two new reference values:

  1. Baseline Value ($V^\emptyset$): This represents the expected return if the environment continues to evolve without the AI agent. Crucially, this includes ongoing human activity. If we’ve already built a city, the baseline assumes the city stays built (and maybe even improves).
  2. Orthogonal Value ($V^\perp$): This captures the value of a trajectory that is statistically independent of human preferences. To make this rigorous, the authors define it as the value of the "simplest" possible trajectory distribution (measured by minimum description length or Kolmogorov complexity).

$\eta$-Catastrophic Performance: The authors define performance $V$ as $\eta$-catastrophic if it falls significantly below the baseline toward the orthogonal value: $$V \leq \eta V^\perp + (1 - \eta)V^\emptyset$$ where $\eta \in [0, 1]$ represents the degree of catastrophe.

Datasets

The paper does not use large-scale empirical datasets. Instead, it employs two illustrative environments to validate the theory:

  • 2D Gridworld: A toy environment where "seek" and "avoid" states are randomly placed. This allows for exact mathematical calculation of baseline and orthogonal values.
  • Minecraft (Conceptual): Used as a high-dimensional example to argue why "simple" distributions (like random block placement) are intuitively catastrophic to humans who want to build complex structures like cities.

Key Insights and Ideas

  • The Rainforest Analogy: A catastrophe isn't always intentional. Humans cutting down a rainforest aren't trying to torture the local wildlife; they are pursuing a goal (timber/land) that is simply unrelated to the animals' welfare. The result is catastrophic for the ecosystem, yet still "better" than if a malicious agent were intentionally maximizing animal suffering.
  • Simplicity is Dangerous: In complex worlds, states that are "easy to describe" (like a vacuum, a uniform distribution of atoms, or a world of random Minecraft blocks) are almost never the states humans find valuable. Therefore, an AI that drifts toward "simple" or "orthogonal" trajectories is naturally destroying the complex, high-value structures humans have built.
  • Baseline as a Threshold: The baseline value provides a clear boundary. Any performance below the baseline is "detrimental," but the orthogonal value gives us a scale to measure how detrimental.

Key Results

The authors demonstrate their framework using a Gridworld example (Figure 1). In this setup, the baseline value is positive because human activity (modeled as the environment's natural drift) tends to move the state toward rewards. The orthogonal value, being unrelated to these rewards, is zero.

2D Gridworld Example showing transition dynamics and the reward landscape. The contemporary state $S_0$ is shown near a high-reward region $S^+$, reflecting prior human optimization.

A key theoretical result (Proposition 4) proves that "simple" distributions cannot concentrate their probability mass on "rare" (complex/desirable) elements. This provides a formal mathematical justification for why an AI that ignores human preferences will almost certainly produce a catastrophic drop in value.

Limitations and Future Work

  • Language Dependence: The definition of "simplicity" depends on the choice of programming language used to measure complexity. While the authors argue this is robust across common languages like Python, it remains a theoretical dependency.
  • Ex Ante Nature: The definition is ex ante, meaning it evaluates the expected performance based on the designer's uncertainty before deployment. It doesn't necessarily categorize a single realized trajectory as catastrophic after the fact.
  • Future Directions: The authors suggest further research into why simple states in our physical world are so rarely desirable to humans—a deep question linking physics, evolution, and AI safety.

Summary updated

Abstract

A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.
BibTeX
@misc{marklund2026definition,
  author = {Henrik Marklund and Alex Infanger and Benjamin Van Roy},
  title = {{A Definition of Catastrophic Performance}},
  year = {2026},
  eprint = {2611.06219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06219},
}
Figure