Biological evolution is a generative engine of unbounded complexity. Over billions of years, it has continuously produced novel structures, behaviors, and ecosystems without ever hitting a final equilibrium or running out of "content."
In contrast, artificial evolutionary systems and reinforcement learning (RL) agents routinely hit a complexity ceiling. Whether simulating artificial life or training neural networks to solve complex tasks, these systems almost always experience a diversity plateau or evolutionary stagnation. After an initial burst of innovation, progress grinds to a halt: a single dominant strategy or pattern monopolizes the environment, or the system fully exhausts the strategic possibilities of its domain.
To build systems capable of continuous, open-ended discovery—a prerequisite for true Artificial Superhuman Intelligence (Hughes et al.)—we must understand why artificial systems plateau and how we can engineer mechanisms to sustain perpetual innovation.
The Stagnation Problem: Why Artificial Evolution Plateaus
In computational simulations, evolution frequently stalls. As synthesized by Packard et al., artificial life environments regularly reach stable states where the generation of novel or complex behaviors ceases.
This stagnation typically manifests in two ways:
- Niche Saturation: In a fixed environment, the space of viable strategies is finite. Once agents discover the globally optimal behaviors, the evolutionary pressure to innovate drops to zero (Baker et al.).
- Competitive Monopolization: In many spatial simulations (such as cellular automata), a single robust, aggressive pattern or species eventually consumes the entire environment, wiping out the diversity required to fuel further selection (Hamon et al.).
To break through this ceiling, researchers have isolated three core pillars that allow a system to sustain open-ended evolution: the relationship between the agent and the world's scale, the dynamics of selection pressure, and the generative capacity of the environment itself.
The "Small Agent, Big World" Principle
Why does an agent stop learning? Intuitively, learning stops when the agent can completely memorize or master its environment. If the environment contains a finite amount of information, an agent with sufficient capacity will eventually compress that information entirely, causing performance to plateau (Kumar et al.).
To force perpetual adaptation, the environment's complexity must infinitely exceed the internal storage and processing capacity of the agent. Kumar et al. formalize this as a "small agent, big world" framework for continual learning. In a truly "big world," the agent can never settle on a static, globally optimal policy because it can only ever experience and represent a fraction of the environment's total state space. Mastery becomes a moving target.
Coevolution and Structural Complexification
One of the most elegant ways to build a "big world" without manually designing infinite content is to make the environment reactive through multi-agent interactions.
When agents compete against one another, the "environment" is no longer a static landscape of obstacles; it is a dynamic web of competing strategies. As demonstrated by Baker et al. in multi-agent hide-and-seek environments, this interaction creates an autocurriculum. When the hiders invent a new strategy (like using tools to lock doors), they fundamentally alter the fitness landscape for the seekers, forcing them to innovate a counter-strategy (like using ramps to jump over walls).
However, an arms race will still plateau if the agents are optimizing within a fixed, rigid architecture. To sustain the arms race indefinitely, the agents must be allowed to complexify their own physical or computational structures.
[ Simple Agent ] ---> Optimizes Parameters ---> [ Strategic Plateau ]
|
v (Structural Complexification)
[ Added Node/Link ] ---> Unlocks New Niches ---> [ Continued Innovation ]
Stanley et al. demonstrated this principle using the NEAT algorithm. By starting with minimal neural network topologies and incrementally adding nodes and connections over generations, evolution does not merely optimize existing behaviors—it introduces entirely new degrees of freedom. This structural growth allows agents to exploit novel strategic niches that were literally impossible to represent in simpler architectures.
Generative Open-Endedness via Foundation Models
While multi-agent autocurricula can drive impressive behavioral emergence, they eventually saturate if the physical environment lacks sufficient expressive power (Baker et al.). To combat this, recent frameworks leverage Large Language Models (LLMs) as open-ended environment generators.
Instead of relying on rigid procedural generation rules, algorithms like LLM-POET (Aki et al.) use LLMs to continuously mutate and generate novel training environments. Because LLMs possess vast world knowledge and semantic understanding, they can generate a diverse, open-ended sequence of challenges that human designers or simple randomized algorithms could not easily conceive.
This generative bootstrapping can also be applied directly to abstract problem-solving. The DéjàQ framework (Röpke et al.) jointly evolves a population of synthetic mathematical problems alongside the training of an LLM. Crucially, the authors select new problems based on learnability—filtering for tasks that are just beyond the model's current capabilities. By dynamically generating problems at the frontier of what the model can learn, the system maintains a continuous training signal, preventing the performance plateaus common to fixed datasets.
Divergent Selection: Rewarding Novelty Over Goals
Even with an infinite environment, evolution can fail if the selection mechanism is purely goal-oriented. Traditional optimization rewards agents for getting closer to a predefined objective. However, in complex landscapes, the stepping stones to a major breakthrough often look completely unrelated to the final goal. Forcing a population to converge on an objective prematurely destroys the radical variations needed for long-term innovation.
To solve this, Lehman et al. argue for divergent selection—abandoning objective functions entirely and instead selecting individuals based on their novelty and diversity.
Objective Search: [Start] -----------------------> [Goal] (Stalls at local optima)
Divergent Search: [Start] ---> [Novelty A] ---> [Novelty B] ---> [Breakthrough]
By rewarding agents simply for doing something different from the rest of the population, Novelty Search fundamentally alters the evolutionary landscape. Doncieux et al. demonstrate that this constant push away from the crowd creates an inevitable pressure toward evolvability. In a novelty-seeking regime, individuals that are highly capable of generating diverse, viable offspring are naturally favored, ensuring that the population maintains its capacity to innovate even within physically restricted environments.
Self-Organization from Minimal Substrates
At the most fundamental level, open-ended evolution must be able to emerge autonomously from simple, decentralized rules without a centralized supervisor or explicit fitness functions (Werfel et al.; Trianni et al.).
To study this, researchers design minimal artificial chemistries grounded in formal logic. For instance, the "Combinatory Chemistry" framework developed by Kruszewski et al. simulates a "soup" of basic computational building blocks. Under simple interaction rules, complex, self-maintaining, and self-reproducing metabolisms emerge spontaneously from a blank slate.
However, these minimal substrates highlight the ultimate frontier of the field: the transition from simple self-reproduction to adaptive evolution. As noted by Kruszewski et al., while artificial chemistries can easily produce structures that replicate, getting those structures to autonomously transition into increasingly complex, adaptive forms over successive generations remains an open challenge.
To detect and encourage these transitions automatically, researchers use neural networks to measure complexity growth. Cisneros et al. demonstrate that by tracking how well a local prediction model generalizes over time within a cellular automaton, one can automatically flag the emergence of complex, self-organizing structures (like gliders). Similarly, Hamon et al. show that curiosity-driven AI agents can be used to autonomously navigate these vast, unguided chemical spaces, discovering the rare, life-like configurations that serve as the building blocks for true open-endedness.
By combining massive, uncompressible environments (Kumar et al.), structural complexification (Stanley et al.), and divergent selection metrics (Lehman et al.), modern AI research is moving away from static optimization and toward systems that generate their own intelligence indefinitely.
Key Foundational Work
Competitive Coevolution through Evolutionary Complexification (2004) — contributed 1 concept elements to this direction.
Validation of evolutionary activity metrics for long-term evolutionary dynamics (2005) — contributed 1 concept elements to this direction.