My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control

Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, Shimon Whiteson

TL;DR

This paper challenges the long-standing assumption that physical limb connections are the best way to structure Graph Neural Networks (GNNs) for robotic control. By replacing these sparse, "morphological" graphs with a fully-connected Transformer architecture called AMORPHEUS, the authors demonstrate significantly better performance and sample efficiency across diverse multitask reinforcement learning benchmarks.

What Problem Does This Paper Address?

Multitask Reinforcement Learning (MTRL) aims to train a single policy that can handle multiple different tasks. However, most MTRL methods struggle with "incompatible" environments—scenarios where agents have different numbers of limbs or sensors (e.g., training one policy to control both a four-legged "Cheetah" and a two-legged "Walker").

To solve this, researchers typically use Graph Neural Networks (GNNs) because they can process graphs of arbitrary size. The standard practice has been to use the agent's physical morphology (how its limbs are physically attached) to define the graph's edges. The intuition was that this "structural inductive bias" helps the model understand the robot's body. This paper asks a critical question: Does this morphological information actually help, or is it a "cage" that restricts efficient learning?

Methods

The authors first perform a series of ablations on state-of-the-art GNN-based controllers (SMP and NerveNet). They test these models using the "correct" physical morphology versus arbitrary structures like a "star" graph (all limbs connected only to the torso) or a "line" graph (limbs connected in a single chain).

Finding that the specific graph structure mattered very little, they propose AMORPHEUS, a Transformer-based architecture. Unlike standard GNNs that pass messages only between connected limbs, AMORPHEUS treats the agent as a fully connected graph.

Figure 2: AMORPHEUS architecture. Lines with squares at the end denote concatenation. Arrows going separately through encoder and decoder denote that rows of the input matrix are processed independently as batch elements. Dashed arrows denote message-passing in a transformer block. The diagram depicts the policy network, the critic has an identical architecture, with the decoder outputs interpreted as value function values.

Key technical details include:

  • Encode-Process-Decode: Each limb's features are encoded independently, processed via self-attention layers, and decoded into joint torques.
  • Global Attention: By using Transformers, every limb can "attend" to every other limb in a single step, bypassing the need for difficult-to-learn multi-hop communication required by sparse physical graphs.
  • Residual Connections: These are used to ensure the decoder doesn't "forget" the original limb features after the attention layers.

Datasets

The study utilizes several established MTRL benchmarks based on the MuJoCo physics engine:

  • Walker++: A collection of walkers with 2 to 7 limbs.
  • Humanoid++: Various humanoid configurations with different active joints.
  • Cheetah++: Cheetah-like robots with varying numbers of legs and segments.
  • Walkers (Wang et al., 2018): A diverse set including Ostrich, HalfCheetah, FullCheetah, Walker, and Hopper agents.

Key Insights and Ideas

  • Morphology as a Bottleneck: In sparse physical graphs, information from a foot might take several "hops" to reach a distant arm. The authors hypothesize that the difficulty of learning this multi-hop communication outweighs any benefit provided by the physical structure.
  • Structure is Overrated: The ablation studies (Figure 1) show that a simple "star" topology often performs as well as, or even better than, the actual physical morphology. This suggests that GNNs aren't actually exploiting the limb-connection information as previously believed.
  • Dynamic Morphology: While AMORPHEUS starts with a fully connected graph, its attention mechanism allows it to learn which limbs need to communicate at any given moment. The authors observed cyclic attention patterns that synchronize with the agent's gait, suggesting the model learns a "functional morphology" that changes during movement.

Key Results

AMORPHEUS consistently and substantially outperforms GNN-based methods across all tested benchmarks.

Comparison of AMORPHEUS vs SMP across various MTRL benchmarks. AMORPHEUS (purple) shows higher returns and faster learning.
  • Superior Performance: In the Walker++ and Humanoid++ tasks, AMORPHEUS reaches higher average returns much faster than the SMP baseline.
  • Stability: In the Cheetah++ benchmark, where SMP is known to be unstable, AMORPHEUS learns successfully and maintains high performance.
  • Generalization: Preliminary results in Table 3 suggest that AMORPHEUS also has strong potential for zero-shot generalization to new, unseen robot morphologies.

Limitations and Future Work

  • Computational Complexity: Transformers have a quadratic ($O(N^2)$) complexity relative to the number of nodes. While not an issue for robots with a dozen limbs, it could become a bottleneck for agents with hundreds of segments or sensors.
  • RL Algorithm Focus: The paper focuses primarily on architecture. The authors note that the underlying RL training loop (e.g., how gradients are averaged across tasks) could be further optimized independently of the Transformer structure.
  • Interpretability: While attention masks offer some insight, fully understanding the "functional structure" learned by the Transformer remains a complex task.

Summary updated

Abstract

Multitask Reinforcement Learning is a promising way to obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited to compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address incompatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based continuous control uses the physical morphology of the agent to construct the input graph, i.e., encoding limb features as node labels and using edges to connect the nodes if their corresponded limbs are physically connected. In this work, we present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose Amorpheus, a transformer-based approach. Further results show that, while Amorpheus ignores the morphological information that GNNs encode, it nonetheless substantially outperforms GNN-based methods that use the morphological information to define the message-passing scheme.
BibTeX
@inproceedings{kurin2021my,
  author = {Vitaly Kurin and Maximilian Igl and Tim Rocktäschel and Wendelin Boehmer and Shimon Whiteson},
  title = {{My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=N3zUDGN5lO},
}
Figure