TL;DR
This paper challenges the long-standing assumption that physical limb connections are the best way to structure Graph Neural Networks (GNNs) for robotic control. By replacing these sparse, "morphological" graphs with a fully-connected Transformer architecture called AMORPHEUS, the authors demonstrate significantly better performance and sample efficiency across diverse multitask reinforcement learning benchmarks.
What Problem Does This Paper Address?
Multitask Reinforcement Learning (MTRL) aims to train a single policy that can handle multiple different tasks. However, most MTRL methods struggle with "incompatible" environments—scenarios where agents have different numbers of limbs or sensors (e.g., training one policy to control both a four-legged "Cheetah" and a two-legged "Walker").
To solve this, researchers typically use Graph Neural Networks (GNNs) because they can process graphs of arbitrary size. The standard practice has been to use the agent's physical morphology (how its limbs are physically attached) to define the graph's edges. The intuition was that this "structural inductive bias" helps the model understand the robot's body. This paper asks a critical question: Does this morphological information actually help, or is it a "cage" that restricts efficient learning?
Methods
The authors first perform a series of ablations on state-of-the-art GNN-based controllers (SMP and NerveNet). They test these models using the "correct" physical morphology versus arbitrary structures like a "star" graph (all limbs connected only to the torso) or a "line" graph (limbs connected in a single chain).
Finding that the specific graph structure mattered very little, they propose AMORPHEUS, a Transformer-based architecture. Unlike standard GNNs that pass messages only between connected limbs, AMORPHEUS treats the agent as a fully connected graph.
Key technical details include:
- Encode-Process-Decode: Each limb's features are encoded independently, processed via self-attention layers, and decoded into joint torques.
- Global Attention: By using Transformers, every limb can "attend" to every other limb in a single step, bypassing the need for difficult-to-learn multi-hop communication required by sparse physical graphs.
- Residual Connections: These are used to ensure the decoder doesn't "forget" the original limb features after the attention layers.
Datasets
The study utilizes several established MTRL benchmarks based on the MuJoCo physics engine:
- Walker++: A collection of walkers with 2 to 7 limbs.
- Humanoid++: Various humanoid configurations with different active joints.
- Cheetah++: Cheetah-like robots with varying numbers of legs and segments.
- Walkers (Wang et al., 2018): A diverse set including Ostrich, HalfCheetah, FullCheetah, Walker, and Hopper agents.
Key Insights and Ideas
- Morphology as a Bottleneck: In sparse physical graphs, information from a foot might take several "hops" to reach a distant arm. The authors hypothesize that the difficulty of learning this multi-hop communication outweighs any benefit provided by the physical structure.
- Structure is Overrated: The ablation studies (Figure 1) show that a simple "star" topology often performs as well as, or even better than, the actual physical morphology. This suggests that GNNs aren't actually exploiting the limb-connection information as previously believed.
- Dynamic Morphology: While AMORPHEUS starts with a fully connected graph, its attention mechanism allows it to learn which limbs need to communicate at any given moment. The authors observed cyclic attention patterns that synchronize with the agent's gait, suggesting the model learns a "functional morphology" that changes during movement.
Key Results
AMORPHEUS consistently and substantially outperforms GNN-based methods across all tested benchmarks.
- Superior Performance: In the Walker++ and Humanoid++ tasks, AMORPHEUS reaches higher average returns much faster than the SMP baseline.
- Stability: In the Cheetah++ benchmark, where SMP is known to be unstable, AMORPHEUS learns successfully and maintains high performance.
- Generalization: Preliminary results in Table 3 suggest that AMORPHEUS also has strong potential for zero-shot generalization to new, unseen robot morphologies.
Limitations and Future Work
- Computational Complexity: Transformers have a quadratic ($O(N^2)$) complexity relative to the number of nodes. While not an issue for robots with a dozen limbs, it could become a bottleneck for agents with hundreds of segments or sensors.
- RL Algorithm Focus: The paper focuses primarily on architecture. The authors note that the underlying RL training loop (e.g., how gradients are averaged across tasks) could be further optimized independently of the Transformer structure.
- Interpretability: While attention masks offer some insight, fully understanding the "functional structure" learned by the Transformer remains a complex task.
Summary updated