Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent

Wadhah Zai El Amri, Luca Hermes, Malte Schilling

TL;DR

This paper introduces a novel control architecture for legged robots that combines hierarchical and decentralized deep reinforcement learning. By splitting control into high-level "planners" and decentralized low-level "executors" (one per leg), the authors achieve faster learning, significantly better energy efficiency, and the ability to solve complex, sparse-reward tasks that traditional centralized models fail to master.

What Problem Does This Paper Address?

Controlling a four-legged robot is a high-dimensional nightmare. Most current Reinforcement Learning (RL) approaches use a centralized controller—a single "brain" that receives all sensor data and outputs all joint commands. While functional, this lacks the modularity found in nature. In biological systems (like insects), movement is both hierarchical (the brain decides where to go, while local circuits handle how to step) and decentralized (individual legs often respond to local sensory feedback).

The authors address three main challenges:

  1. Learning Efficiency: Centralized models struggle with the "curse of dimensionality" as the number of joints increases.
  2. Energy Efficiency: Standard RL often produces jittery, power-hungry movements.
  3. Generalization/Transfer: It is difficult to take a robot that has learned to walk and teach it a complex new task (like navigating a maze) without "catastrophic forgetting" or starting from scratch.

Methods

The researchers benchmarked five different architectures using the Proximal Policy Optimization (PPO) algorithm:

  • Flat Architectures: Centralized (one brain) and Decentralized (four independent leg controllers).
  • Hierarchical Architectures:
    • C_C: Centralized High-Level Controller (HLC) + Centralized Low-Level Controller (LLC).
  • C_D: Centralized HLC + Decentralized LLC (one controller per leg).
  • D_D: Decentralized HLC + Decentralized LLC.

The Hierarchical Split: The HLC operates at a lower frequency, processing global data (like goal position) to produce a "latent command." The LLC operates at a high frequency, taking that latent command plus local joint data to produce actual motor torques. In the decentralized versions, the information flow is restricted: each leg controller only sees its own joint states, forcing the system to learn modular coordination.

Datasets & Experimental Setup

The study used the MuJoCo physics engine with a modified Ant-v3 model (weighted to 10x its default mass to simulate a real-world robot). They designed three benchmark tasks:

  1. Task 1 (Navigate to Goal): Continuous reward based on distance to a random target.
  2. Task 2 (Seek Target): Sparse reward; the agent only gets points upon reaching the goal. This tests transfer learning, as hierarchical models used frozen LLCs trained in Task 1.
  3. Task 3 (Maze Navigation): A 15x15m maze requiring memory (implemented via an LSTM layer) and subgoal navigation.

Key Insights and Ideas

  • Decentralization as a Regularizer: By restricting a leg's controller to only see its own data, the authors found the agent learns faster. It prevents the "brain" from over-relying on complex correlations across the whole body, leading to more robust, modular walking primitives.
  • Hierarchy Enables Transfer: Flat architectures (Cen/Dec) completely failed at the sparse-reward tasks (Tasks 2 and 3). Because the hierarchical models could "freeze" their walking skills (LLC) and only train the navigation logic (HLC), they could solve tasks where feedback was rare.
  • The "C_D" Sweet Spot: The architecture with a Centralized HLC and Decentralized LLC (C_D) emerged as the most effective. It provides a single point of strategic decision-making while maintaining the efficiency and modularity of decentralized motor control.

Key Results

  • Energy Efficiency: In Task 1, decentralized hierarchical models (C_D and D_D) were significantly more power-efficient than the flat decentralized model. They produced smoother movements with lower torque requirements.
  • Success in Sparse Rewards: In Task 2 (Sparse Reward), the C_D architecture achieved a 99.6% success rate on flat terrain. In contrast, both flat centralized and flat decentralized models failed to learn the task entirely (0% success).
  • Robustness: All trained architectures showed high success rates when tested on unseen, "bumpy" terrain, but the hierarchical decentralized models maintained better energy efficiency even under these difficult conditions.
  • Maze Solving: Only the hierarchical models were able to solve the maze task effectively, with the C_D architecture reaching the highest return values.

Limitations and Future Work

The authors identify several areas for expansion:

  • Simulation vs. Reality: The work is currently limited to the MuJoCo simulator. Real-world physics (latency, sensor noise, friction) may pose additional challenges for decentralized coordination.
  • Fixed Hierarchy: The degree of decentralization was fixed (per leg). Future work could explore "per-joint" decentralization or dynamic hierarchies.
  • Robustness to Failure: A major theoretical benefit of decentralization is resilience to local failure (e.g., a broken leg). The authors suggest testing whether their HLCs can automatically adapt to such hardware failures in future studies.

Summary updated

Abstract

Legged locomotion is widespread in nature and has inspired the design of current robots. The controller of these legged robots is often realized as one centralized instance. However, in nature, control of movement happens in a hierarchical and decentralized fashion. Introducing these biological design principles into robotic control systems has motivated this work. We tackle the question whether decentralized and hierarchical control is beneficial for legged robots and present a novel decentral, hierarchical architecture to control a simulated legged agent. Three different tasks varying in complexity are designed to benchmark five architectures (centralized, decentralized, hierarchical and two different combinations of hierarchical decentralized architectures). The results demonstrate that decentralizing the different levels of the hierarchical architectures facilitates learning of the agent, ensures more energy efficient movements as well as robustness towards new unseen environments. Furthermore, this comparison sheds light on the importance of modularity in hierarchical architectures to solve complex goal-directed tasks. We provide an open-source code implementation of our architecture (https://github.com/wzaielamri/hddrl).
BibTeX
@inproceedings{amri2022hierarchical,
  author = {Wadhah Zai El Amri and Luca Hermes and Malte Schilling},
  title = {{Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent}},
  booktitle = {LOD},
  year = {2022},
  doi = {10.1007/978-3-031-25891-6\_20},
}

Loading related content...

Figure