Abstract
Contemporary sensorimotor learning approaches typically start with an existing complex agent (e.g., a robotic arm), which they learn to control. In contrast, this paper investigates a modular co-evolution strategy: a collection of primitive agents learns to dynamically self-assemble into composite bodies while also learning to coordinate their behavior to control these bodies. Each primitive agent consists of a limb with a motor attached at one end. Limbs may choose to link up to form collectives. When a limb initiates a link-up action, and there is another limb nearby, the latter is magnetically connected to the 'parent' limb's motor. This forms a new single agent, which may further link with other agents. In this way, complex morphologies can emerge, controlled by a policy whose architecture is in explicit correspondence with the morphology. We evaluate the performance of these dynamic and modular agents in simulated environments. We demonstrate better generalization to test-time changes both in the environment, as well as in the structure of the agent, compared to static and monolithic baselines.
Introduction
In traditional robotics and reinforcement learning (RL), the standard paradigm involves training a fixed agent—such as a humanoid or a robotic arm—to perform a specific task. While successful, this monolithic approach faces significant challenges in scaling and flexibility. As robotic complexity increases, the state and action spaces grow exponentially, complicating the training process. Furthermore, policies trained for a specific morphology are inherently rigid; a controller for a quadruped cannot easily adapt to the addition or loss of a limb.
Inspired by biological multicellularity, where individual cells cooperate to form complex, adaptive organisms, this research proposes a shift toward modularity. By utilizing a modular policy that mirrors the physical structure of the agent, we can move away from "one brain for one fixed body" toward a universal modular policy capable of controlling diverse and dynamic morphologies.
Modular Co-Evolution Strategy
The proposed framework relies on two primary components: a primitive limb agent and a Dynamic Graph Network (DGN) policy.
The Primitive Agent
Each module, or "limb," is a simple cylinder equipped with a motor at its "parent" end and a free "child" end. These limbs can perform discrete actions to "link" or "unlink." When a limb is within magnetic range of another, they dock to form a joint with three degrees of freedom. This allows for the emergence of complex, tree-structured assemblies from simple building blocks.
Dynamic Graph Networks (DGN)
The "brain" of the collective is implemented as a Dynamic Graph Network. Instead of a single centralized neural network, every limb executes the same small neural network with shared parameters. These networks coordinate by passing messages to their physical neighbors in the assembly graph. Because the policy's structure is in explicit correspondence with the robot's physical morphology, the controller automatically scales as limbs are added or removed.
Experimental Setup
The system was evaluated in a custom Unity ML simulation environment on two primary tasks:
- Standing Up: Limbs start scattered and must assemble into the tallest possible tower. The reward is determined by the height of the highest point in the assembly.
- Locomotion: Limbs must assemble and travel as far as possible along the X-axis.
Training was conducted using Proximal Policy Optimization (PPO), with rewards shared among all limbs within a connected component. This setup encourages the emergence of an internal curriculum, where limbs first learn simple coordination before tackling complex multi-limb movements.
Results and Generalization
The modular approach demonstrated significant advantages over monolithic baselines in both scaling and robustness. While monolithic networks struggled to reach optimal performance as the number of limbs increased, the DGN-based agents maintained high performance regardless of limb count.
Zero-Shot Generalization
A key strength of the modular policy is its ability to generalize to novel configurations and environments without additional training. For instance, a policy trained on 6 limbs for the "Standing Up" task successfully controlled 12 limbs at test time, achieving a higher score (19,796) than during training (17,518) by leveraging the additional hardware.
Furthermore, the agents exhibited remarkable robustness to environmental perturbations:
- Terrain Adaptability: In locomotion tasks, DGN agents retained over 70% of their training performance when navigating hurdles and stairs, whereas monolithic baselines often failed entirely.
- Environmental Forces: In the standing task, the modular policy retained 96% of its performance in a simulated "underwater" environment with increased drag and buoyancy, demonstrating the inherent stability of modular coordination.
Discussion
This work highlights that modularity in both software (DGN) and hardware (self-assembling limbs) is essential for creating adaptable robotic systems. By treating morphology as an action, agents can discover beneficial shapes for specific tasks dynamically.
However, several avenues for future work remain. The current implementation is limited to tree-structured assemblies; extending this to cyclic structures would require more complex message-passing algorithms like loopy belief propagation. Additionally, exploring heterogeneous modules with different sensors or actuators could further enhance the capabilities of these self-assembling collectives. Finally, while the policy is decentralized, the reward signal remains global, presenting a challenge for fully decentralized learning in the future.
Summary updated