Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity

Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, Alexei A. Efros

Introduction

In traditional reinforcement learning (RL), researchers typically start with a fixed, complex agent—such as a humanoid or a quadruped—and train a "monolithic" neural network to control it. While effective in controlled settings, this approach is notoriously brittle. If the robot’s body changes (e.g., losing a leg) or if it is placed in a radically different environment, the policy often fails completely.

Nature, however, solves this problem through modularity. Multicellular organisms evolved by combining simple cells into complex, adaptable structures capable of surviving chaotic environments. This paper investigates whether we can replicate this biological modularity in artificial agents. By co-evolving morphology (the physical shape) and control (the software), the researchers aim to create agents capable of "zero-shot" generalization—handling entirely new scenarios and body shapes they never encountered during training.

Methods: Dynamic Self-Assembly

The authors introduce a framework where the agent is not a single, static entity, but rather a collection of primitive limbs that learn to dynamically self-assemble.

  1. Agent Structure: Each primitive agent is a simple cylinder with a motor at one end (the "parent-end") and a free "child-end." Limbs can use magnetic forces to link up with one another.
  2. Morphology as an Action: Unlike static robots, these limbs treat linking and unlinking as dynamic actions within an RL episode. They can literally change their shape on the fly to better solve a given task.
Overview of the modular framework: from scattered limbs in the environment to assembled agents controlled by a structurally corresponding modular policy.
  1. Dynamic Graph Networks (DGN): The core technical innovation is representing the control policy as a Graph Neural Network (GNN), where each node corresponds to a physical limb.
  2. Modularity: All limbs share the exact same policy parameters.
  3. Structural Correspondence: The topology of the neural network perfectly matches the physical arrangement of the limbs.
  4. Message Passing: Limbs pass learned vectors (messages) to their neighbors to coordinate movement. If the robot's physical shape changes, the network's message-passing flow updates instantly to match.
  5. Optimization: The system is trained using Proximal Policy Optimization (PPO). While control is decentralized, the reward is shared among all limbs in a connected assembly, ensuring they work toward a common goal.

Training Environments and Tasks

Because standard RL benchmarks (like MuJoCo) do not support dynamically changing morphologies, the authors built a custom environment using the Unity ML framework. They focused on two primary tasks:

  • Standing Up: Maximize the height of the highest point of the assembled structure.
  • Locomotion: Move as far as possible along the X-axis.

To test robustness, they created several novel generalization terrains, including bumpy tiles, stairs, hurdles, gaps, "underwater" conditions (increased drag and buoyancy), and "windy" conditions (random forces applied to the limbs).

Key Insights

  • The Power of Shared Parameters: Because every limb uses the exact same "brain" (shared weights), the policy is agnostic to the total number of limbs attached. It learns local rules of coordination that scale globally.
  • Morphological Curriculum: Training naturally begins with limbs scattered on the ground. They learn to form small groups first, then larger ones. This acts as an organic curriculum, where the agent masters simple shapes before attempting complex ones.

Key Results

The DGN approach significantly outperformed monolithic baselines—where one large network processes all inputs simultaneously—in both training speed and zero-shot generalization.

Zero-shot generalization across novel environments and morphologies for standing up and locomotion tasks.
  • Limb Generalization: A policy trained on 6 limbs to "stand up" was tested zero-shot on 12 limbs. It achieved a score of 19,796 (113% of its training performance), effectively leveraging the extra limbs to build a much taller tower. Monolithic baselines cannot even run on a different number of limbs without being entirely redesigned and retrained.
  • Environmental Robustness: In locomotion tasks on novel "Hurdles," the DGN maintained 73% of its training performance. In contrast, monolithic baselines failed completely, often resulting in negative scores (indicating an inability to move forward).
  • Zero-Shot Adaptation: The modular agents successfully navigated "underwater" and "windy" conditions with minimal performance drops, proving that structural modularity breeds inherent robustness to external physical perturbations.

Limitations and Future Work

While promising, the current framework has limitations. The implementation currently restricts morphologies to tree structures, meaning limbs cannot form closed loops or rings. Extending the system to general graphs using iterative message passing (like loopy belief propagation) is a key area for future work. Additionally, the magnetic connection is a simplified abstraction; real-world physical docking involves complex alignment and hardware constraints. Finally, while control is decentralized, the reward signal remains global. Future research could explore how purely local rewards might lead to similar emergent, cooperative behaviors.

Summary updated

Abstract

Contemporary sensorimotor learning approaches typically start with an existing complex agent (e.g., a robotic arm), which they learn to control. In contrast, this paper investigates a modular co-evolution strategy: a collection of primitive agents learns to dynamically self-assemble into composite bodies while also learning to coordinate their behavior to control these bodies. Each primitive agent consists of a limb with a motor attached at one end. Limbs may choose to link up to form collectives. When a limb initiates a link-up action and there is another limb nearby, the latter is magnetically connected to the 'parent' limb's motor. This forms a new single agent, which may further link with other agents. In this way, complex morphologies can emerge, controlled by a policy whose architecture is in explicit correspondence with the morphology. We evaluate the performance of these dynamic and modular agents in simulated environments. We demonstrate better generalization to test-time changes both in the environment, as well as in the structure of the agent, compared to static and monolithic baselines. Project videos and source code are provided in the supplementary material.
BibTeX
@inproceedings{pathak2019learning,
  author = {Deepak Pathak and Christopher Lu and Trevor Darrell and Phillip Isola and Alexei A. Efros},
  title = {{Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=0VyS9Rr3Sx},
}
Figure