Learning a Distributed Hierarchical Locomotion Controller for Embodied Cooperation

Chuye Hong, Kangyao Huang, Huaping Liu

Introduction

Embodied cooperation—where multiple robots must coordinate their physical movements to achieve a shared goal—represents a significant challenge in robotics. Unlike high-level coordination (e.g., swarm formation) or static manipulation, embodied cooperation requires robots to manage complex whole-body locomotion while simultaneously responding to the physical forces and intentions of their teammates. Traditional centralized approaches often fail to scale, while decentralized methods frequently lack the temporal reasoning necessary for sequential tasks. This paper introduces a distributed, three-layer hierarchical reinforcement learning (HRL) framework designed to enable scalable and intelligent cooperation among legged robots.

Distributed Hierarchical Framework

The core of the proposed system is a three-layer controller that separates high-level perception and planning from low-level motor execution. This modularity allows the system to handle complex strategies without sacrificing the stability of individual robot locomotion.

Overview of the distributed HRL architecture featuring Upper (UL), Middle (ML), and Lower (LL) layer modules.

The framework consists of the following components:

  1. Upper Layer (UL - Perception): This layer processes exteroceptive observations, such as terrain heightmaps and the relative position of the nearest neighbor. By focusing only on the closest peer, the input size remains constant, ensuring the system is scalable to any number of agents.
  2. Middle Layer (ML - Planning): Acting as the "brain," the ML utilizes a Recurrent Neural Network (RNN) to maintain a hidden state. This spatiotemporal memory is crucial for tasks requiring causal inference or sequential coordination, such as waiting for a teammate to pass. It generates high-level commands for the execution layer.
  3. Lower Layer (LL - Execution): This is a pre-trained, frozen locomotion controller responsible for maintaining balance and executing leg movements. By decoupling locomotion from high-level strategy, the training process is significantly simplified.

The entire system is trained using Independent Proximal Policy Optimization (IPPO) combined with a curriculum-based reward structure that gradually introduces complexity.

Experimental Evaluation

To evaluate the framework, the researchers developed three distinct benchmark scenarios within the NVIDIA IsaacSim environment, each testing a different facet of cooperation.

Benchmark scenarios used for evaluation: (a) Cooperative Transport, (b) Corridor Crossing, and (c) Ravine Bridging.
  • Cooperative Transport: A group of quadruped "Ant" robots must collectively push a heavy, large-scale object to a target location. This task tests the agents' ability to handle external physical loads and coordinate their pushing directions.
  • Corridor Crossing: Robots must navigate through a narrow passage that only allows one agent at a time. This scenario evaluates "social" coordination and the ability to take turns.
  • Ravine Bridging: This heterogeneous task requires some agents to push a block into a gap, creating a bridge for other agents to cross. It tests complex role-playing and sequential task completion.

Results and Key Insights

The HRL framework demonstrated significant performance gains over non-hierarchical and memory-less baselines. In the Cooperative Transport task, the system achieved a 76.8% success rate, whereas models without hierarchy or memory failed almost entirely. The Corridor Crossing task saw an 88.4% success rate, with agents naturally developing conciliatory behaviors—waiting for others to pass—thanks to the RNN-based memory in the Middle Layer.

The Ravine Bridging task, despite its extreme complexity, reached a 50.4% success rate. Furthermore, the decentralized nature of the controller allowed for zero-shot transfer; a model trained with four agents could be deployed with two, three, or five agents without additional training. In transport tasks, performance actually improved as more agents were added, demonstrating effective workload distribution.

Conclusion

This research highlights that hierarchy and memory are essential for complex embodied cooperation. By isolating locomotion from high-level planning and utilizing recurrent states for temporal reasoning, the proposed distributed HRL framework achieves scalable and robust performance across diverse tasks. Future work will focus on reducing the reliance on manual reward shaping—potentially through the use of Large Language Models for high-level guidance—and addressing the challenges of sim-to-real transfer to deploy these controllers on physical hardware.

Summary updated

Abstract

In this work, we propose a distributed hierarchical locomotion control strategy for whole-body cooperation and demonstrate the potential for migration into large numbers of agents. Our method utilizes a hierarchical structure to break down complex tasks into smaller, manageable sub-tasks. By incorporating spatiotemporal continuity features, we establish the sequential logic necessary for causal inference and cooperative behaviour in sequential tasks, thereby facilitating efficient and coordinated control strategies. Through training within this framework, we demonstrate enhanced adaptability and cooperation, leading to superior performance in task completion compared to the original methods. Moreover, we construct a set of environments as the benchmark for embodied cooperation.
BibTeX
@inproceedings{hong2024learning,
  author = {Chuye Hong and Kangyao Huang and Huaping Liu},
  title = {{Learning a Distributed Hierarchical Locomotion Controller for Embodied Cooperation}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=NCnplCf4wo},
}
Figure