Abstract
Many real-world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration, except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.
Introduction
This research from OpenAI addresses a fundamental challenge in AI alignment: how do we train AI systems on tasks that are too complex for humans to evaluate directly? By recursively breaking down difficult problems into simpler ones, the authors demonstrate that an agent can "bootstrap" its way to high-level competence without requiring an external reward function.
The Problem: Beyond Human-Scale Tasks
Most machine learning relies on a clear training signal: either an algorithmic one (like the score in a game) or a human one (like a person labeling an image). However, we are approaching a frontier of "beyond human scale" tasks—such as managing national economic policy or securing massive computer networks—where a single human cannot provide a reliable reward signal because the observation space is too vast or the consequences are too long-term.
When we cannot provide a perfect signal, we often use "proxies" (e.g., user clicks instead of user satisfaction). As agents become more powerful, they tend to "game" these proxies, leading to pathological behavior known as Goodhart’s Law: "When a measure becomes a target, it ceases to be a good measure." This paper seeks a way to supervise agents on complex tasks where the objective is implicit and hard to specify.
Methodology: Iterated Amplification
The core proposal is Iterated Amplification, a training framework where a human expert supervises an AI by breaking complex tasks into simpler sub-tasks that the AI helps solve. The process works in a recursive loop:
- The Agent ($X$): Start with a weak model.
- The Amplified Expert ($Amplify^H(X)$): A human expert ($H$) is asked to solve a hard question ($Q$). Instead of answering it directly, the human breaks $Q$ into several simpler sub-questions ($Q_1, Q_2, ...$). The human then uses copies of the current agent $X$ to answer these sub-questions and aggregates those answers to produce the final result ($A$).
- The Learning Step: The agent $X$ is trained via supervised learning to predict the output of the "amplified" system. Essentially, $X$ tries to become as smart as the human-plus-multiple-copies-of-$X$ team.
- Iteration: As $X$ gets better at answering sub-questions, the "amplified" team becomes even more capable, providing an even stronger training signal for the next version of $X$.
To make this practical, the authors also train a human predictor ($H'$) to imitate how the human decomposes problems, reducing the actual amount of human labor required. The model architecture is based on the Transformer, modified to handle sets of facts rather than just sequences.
Experimental Evaluation
The authors evaluated the method on five synthetic algorithmic tasks designed to mimic complex reasoning:
- Permutation Powering: Computing the result of a permutation applied $k$ times.
- Sequential Assignments: Evaluating a variable defined by a long sequence of intermediate assignments.
- Wildcard Search: Summing function values over inputs matching a specific pattern.
- Shortest Path: Finding the distance or first step between nodes in a directed graph.
- Union Find: Identifying labels in a forest of rooted trees.
These tasks were chosen because they are easy to verify algorithmically but require multi-step "thinking" that can be decomposed into recursive sub-problems.
Key Insights and Results
- Successful Bootstrapping: Iterated Amplification successfully learned to solve all five algorithmic tasks, achieving accuracy nearly identical to standard supervised learning.
- Decomposition Efficiency: While supervised learning required tens of millions of examples to learn the tasks from scratch, the model learned the decompositions using only tens of thousands of "oracle" calls. This suggests that learning how to solve a problem is much cheaper than learning the solution itself.
- Implicit Objectives: There is no "reward" given by the environment. The "goal" of the task is defined entirely by how the human chooses to decompose and recompose the sub-problems.
- The "Human Scale" Bridge: This method provides a theoretical path to supervising "superhuman" AI. If a human can coordinate a team of AIs to perform a task, they can train a single AI to do that task, even if the human couldn't do it alone.
Limitations and Future Work
While promising, the experiments were limited to discrete, algorithmic environments with hard-coded decompositions. Future work must address whether real humans can effectively decompose messy, real-world tasks like writing code or policy. Additionally, the system must be robust to "distribution shift," ensuring the agent can handle the variety of sub-questions generated during recursion.
Summary updated