One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

Nico Bohlinger, Grzegorz Czechmanowski, Maciej Krupka, Piotr Kicki, Krzysztof Walas, Jan Peters, Davide Tateo

TL;DR

The authors introduce URMA (Unified Robot Morphology Architecture), a single, morphology-agnostic neural network policy capable of controlling 16 completely different legged robots—ranging from quadrupeds and hexapods to bipeds and humanoids. By utilizing attention-based encoders and decoders conditioned on physical joint descriptions, URMA learns a generalized locomotion controller that can transfer zero-shot to entirely unseen robots in both simulation and the real world.

What Problem Does This Paper Address?

In deep reinforcement learning (DRL) for legged locomotion, policies are almost always locked to a specific robot. A controller trained for a four-legged Unitree A1 cannot be deployed on an ANYmal, let alone a two-legged humanoid, because their observation and action spaces differ based on the number of joints and limbs.

Traditional Multi-Task Reinforcement Learning (MTRL) attempts to solve this by either padding observation vectors with zeros to match a maximum size or using separate neural network "heads" for each robot. However, these approaches scale poorly, require retraining from scratch for any new robot configuration, and fail to generalize to unseen morphologies. URMA tackles this fundamental "correspondence problem" by creating a single framework that treats different robot bodies as variations of a single underlying control task.

Methods

URMA achieves morphology independence through a clever separation of general state observations and robot-specific joint/foot characteristics:

  1. General Observations: Fixed-size inputs common to all robots, such as angular velocity from an IMU, command velocities, and gravity vectors.
  2. Joint Descriptions & Observations: For every joint, the network receives its current state (position, velocity) alongside a fixed-size description vector containing its physical properties (3D relative position, rotation axis, torque/velocity limits, rotor inertia, etc.).
  3. Attention-Based Encoding: A shared Multilayer Perceptron (MLP) encodes each joint's data. A softmax attention mechanism with a learnable temperature then aggregates these variable-sized joint inputs into a single, fixed-size latent vector. This allows the network to dynamically route relevant joint information regardless of how many joints the robot has.
  4. Universal Morphology Decoding: To output actions, a universal decoder pairs the core network's action latent vector back with each individual joint's description vector to sample joint-specific motor commands.

The entire architecture is trained end-to-end using Proximal Policy Optimization (PPO) combined with a time-dependent penalty curriculum to smoothly shape the robots' gaits over time.

Figure 2: Overview of URMA. Left – Joint observations and descriptions are encoded and com- bined into a single joint latent vector through an attention head. Bottom center – Feet observations and descriptions are encoded in the same way. Top center – Joint latent, feet latent, and general observations are fed through the core network to get the action latent vector. Right – The universal morphology decoder encodes the joint descriptions and pairs them with the action latent vector and the single joint latent vector to produce the action mean and standard deviation for the final action.

Datasets

The policy was trained entirely in the CPU-based MuJoCo physics simulator using a diverse suite of 16 simulated robots across 48 parallel environments:

  • 9 Quadrupeds: Including ANYmal B/C, Google Barkour v0/vB, MAB Silver Badger, Petoi Bittle, and Unitree A1/Go1/Go2.
  • 5 Humanoids: PAL Robotics Talos, Robotis OP3, SoftBank Nao V5, Unitree G1, and Unitree H1.
  • 1 Biped: Agility Robotics Cassie.
  • 1 Hexapod: A custom 18-joint hexapod.

For real-world validation, the policy was deployed on physical hardware using the Unitree A1, MAB Honey Badger, and MAB Silver Badger.

The diverse lineup of 16 simulated robots used for training URMA, alongside real-world deployment platforms.

Key Insights and Ideas

  • Conditioning on Physics, Not Just State: By feeding the network explicit joint parameters (like friction, damping, and gear ratios), the policy doesn't just memorize a gait; it learns to reason about how a joint will react based on its physical constraints.
  • Attention as a Dynamic Router: The attention mechanism acts as a universal translator. It allows the network to automatically adjust its internal representations whether it is balancing a top-heavy humanoid or coordinating a stable six-legged hexapod gait.
  • The Multi-Task Advantage: Counterintuitively, training on 16 diverse robots simultaneously yields higher final performance and better sample efficiency than training individual, specialized policies for each robot from scratch. The shared representation space accelerates learning across different morphologies.

Key Results

  • Superiority Over Baselines: URMA consistently outperformed standard padding and multi-head MTRL baselines in both training efficiency and final reward across the 16-robot fleet.
  • Zero-Shot Transfer: When evaluated on a simulated Unitree A1 withheld from training, URMA achieved immediate, optimal locomotion performance, vastly outperforming single-robot policies trained for the same duration.
  • Out-of-Distribution Generalization: URMA successfully controlled the MAB Silver Badger—a robot featuring an extra spine joint in its trunk and lacking foot sensors—outperforming baselines and maintaining a stable gait through zero-shot transfer and subsequent few-shot fine-tuning.
  • Real-World Deployment: The exact same URMA policy trained in simulation was successfully deployed zero-shot onto three physical quadruped robots, enabling robust walking over pavement, grass, and turf. Crucially, this included the MAB Honey Badger, which was entirely absent from the simulation training set.

Limitations and Future Work

While URMA represents a major step toward a "foundation model" for locomotion, the authors note several limitations:

  • Blind Locomotion: The current framework omits exteroceptive sensors (like cameras or LiDAR), meaning the robots cannot navigate complex obstacles or uneven terrain requiring vision.
  • Zero-Shot Humanoid Transfer: Zero-shot transfer remains highly challenging for humanoids due to their inherent instability and the limited number of bipedal platforms in the training set. Expanding the diversity of the training fleet and incorporating advanced data augmentation are suggested avenues to bridge this gap.

Summary updated

Abstract

Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.
BibTeX
@inproceedings{bohlinger2024one,
  author = {Nico Bohlinger and Grzegorz Czechmanowski and Maciej Krupka and Piotr Kicki and Krzysztof Walas and Jan Peters and Davide Tateo},
  title = {{One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Ef97P71PJE},
}
Figure