In biological history, evolution does not "finish." It does not reach a single optimal organism and stop; instead, it produces an ever-expanding tree of complexity, from single cells to social civilizations. In contrast, most artificial intelligence and artificial life simulations suffer from a "complexity plateau." Whether they are optimizing a neural network's weights or simulating a digital ecosystem, these systems typically reach an equilibrium where innovation ceases and the system stagnates.
The research direction of Open-Ended Evolution (OEE) seeks to bridge this gap. It aims to design systems that continuously generate novel, complex, and "interesting" forms without human intervention.
The Problem of Stagnation
As noted in the proceedings of ALIFE XI, a primary challenge in artificial life is that models eventually reach a state of equilibrium. Vito Trianni and colleagues argue that this lack of open-endedness often stems from a reliance on centralized selection mechanisms or fixed environments that offer a finite number of "niches." Once an agent solves the environment's challenges, there is no further pressure to innovate.
To overcome this, researchers have identified several key mechanisms that allow a system to keep moving.
Complexification: Beyond Parameter Tuning
Early neuroevolution focused on optimizing the weights of a fixed neural architecture. However, Kenneth Stanley and Risto Miikkulainen argued in their work on NEAT that true innovation requires complexification. Instead of just changing how a brain works, evolution must be able to change the structure of the brain itself—adding new neurons and connections over time.
By starting with simple models and incrementally adding complexity, evolution can maintain an "arms race." This principle is extended in ATEP by Michael Beukman, which allows agents to evolve their neural topologies as they encounter increasingly difficult challenges, preventing the structural stagnation that occurs when an agent's "brain size" is fixed.
Co-evolution and the Environment
If the environment is static, evolution will eventually "solve" it. Open-endedness requires a dynamic environment that evolves alongside the agents.
- Emergent Curricula: Systems like PINSKY co-evolve game levels and agent policies. Aaron Dharna and Lisa B. Soros found that the most critical moments in these systems are "stepping stones"—rare transfers where an agent learns a skill in one environment that unexpectedly unlocks progress in a much more difficult one.
- Automated World-Building: OMNI-EPIC, developed by Maxence Faldor and Jeff Clune, uses Large Language Models to autonomously code new environments. By using "interestingness" as a metric, the system generates an infinite stream of novel challenges, from dodgeball to restaurant service, ensuring the agents never run out of things to learn.
Self-Organization in Continuous Substrates
A different approach to OEE focuses on Self-Organization (SO) within cellular automata. Rather than explicitly programming agents, researchers look for life-like structures that emerge from local rules.
Bert Wang-Chak Chan’s Lenia framework provides a continuous "chemical soup" where complex, moving patterns emerge. To drive these patterns toward OEE, Niko Lorantos introduced multi-objective ranking that rewards stability, novelty, and sparsity simultaneously.
The recent Flow-Lenia by Erwan Plantec and Pierre-Yves Oudeyer adds mass conservation to this system. By treating activations as "matter" that flows, the system naturally produces multi-species simulations where speciation and extinction emerge from internal dynamics rather than external fitness functions.
Quality-Diversity and Curiosity
In OEE, the goal is often not to find the best solution, but to find all possible interesting solutions. This is the domain of Quality-Diversity (QD) algorithms.
Maxence Faldor and Antoine Cully developed Leniabreeder, which uses QD to discover a vast ecosystem of patterns. Similarly, Gautier Hamon explores how curiosity-driven algorithms can act as a catalyst for OEE, pushing the search into unexplored regions of the state space to avoid the "diversity plateau" where a single dominant species takes over.
The Biological Arrow of Time
Why does complexity increase at all? Fernando Rosas and Mikhail Prokopenko suggest a theoretical framework based on Turing jumps. They argue that major evolutionary transitions occur when organisms begin to model themselves, creating a self-referential computational loop. This "biological arrow of time" suggests that OEE is not just about adding more parts, but about the emergence of new levels of information hierarchy.
By shifting the focus from solving specific tasks to maintaining a perpetual process of innovation—through complexification, co-evolution, and self-organization—this field moves closer to creating artificial systems that possess the creative power of natural evolution.