Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent

Wadhah Zai El Amri, Luca Hermes, Malte Schilling

Abstract

Legged locomotion is widespread in nature and has inspired the design of current robots. The controller of these legged robots is often realized as one centralized instance. However, in nature, control of movement happens in a hierarchical and decentralized fashion. Introducing these biological design principles into robotic control systems has motivated this work. We tackle the question whether decentralized and hierarchical control is beneficial for legged robots and present a novel decentral, hierarchical architecture to control a simulated legged agent. Three different tasks varying in complexity are designed to benchmark five architectures (centralized, decentralized, hierarchical and two different combinations of hierarchical decentralized architectures). The results demonstrate that decentralizing the different levels of the hierarchical architectures facilitates learning of the agent, ensures more energy efficient movements as well as robustness towards new unseen environments. Furthermore, this comparison sheds light on the importance of modularity in hierarchical architectures to solve complex goal-directed tasks.

Introduction

Controlling a four-legged robot presents a significant challenge in high-dimensional motor control. Most current Deep Reinforcement Learning (DRL) approaches rely on a centralized controller—a single "brain" that processes all sensory inputs and outputs all joint commands simultaneously. While effective in simple environments, these models often struggle with the "curse of dimensionality," high energy consumption, and poor generalization to complex tasks.

In nature, biological systems like insects utilize a different paradigm: movement is both hierarchical and decentralized. A high-level system (the brain) determines the direction and goal, while low-level local circuits (central pattern generators in individual limbs) handle the mechanics of stepping and local sensory feedback. This research investigates whether implementing these biological principles—specifically the combination of hierarchical and decentralized control—can improve the performance and efficiency of simulated legged agents.

Architectural Framework

The researchers benchmarked five distinct architectures using the Proximal Policy Optimization (PPO) algorithm to evaluate the impact of hierarchy and decentralization:

  1. Flat Centralized (Cen): A single controller for all joints.
  2. Flat Decentralized (Dec): Four independent controllers, one for each leg.
  3. Hierarchical Centralized (C_C): A Centralized High-Level Controller (HLC) and a Centralized Low-Level Controller (LLC).
  4. Hierarchical Semi-Decentralized (C_D): A Centralized HLC paired with a Decentralized LLC (one controller per leg).
  5. Hierarchical Fully Decentralized (D_D): A Decentralized HLC and a Decentralized LLC.

The Hierarchical Split

In the hierarchical models, the control task is divided by frequency and scope. The HLC operates at a lower frequency, processing global data (such as goal position) to produce a "latent command." The LLC operates at a higher frequency, translating that latent command and local joint data into specific motor torques. In decentralized versions, information flow is strictly limited: each leg controller only observes its own joint states, forcing the system to learn modular coordination rather than relying on global correlations.

Experimental Setup

The study utilized the MuJoCo physics engine with a modified Ant-v3 model, weighted to ten times its default mass to better simulate real-world robotic dynamics. Three benchmark tasks were designed to test different capabilities:

  • Task 1 (Navigate to Goal): A continuous reward task based on the agent's distance to a random target.
  • Task 2 (Seek Target): A sparse reward task where the agent only receives points upon reaching the goal. This tested transfer learning, as hierarchical models used "frozen" LLCs previously trained in Task 1.
  • Task 3 (Maze Navigation): A 15x15m maze requiring memory (implemented via an LSTM layer) and complex subgoal navigation.

Key Results and Insights

1. Decentralization as a Regularizer

By restricting a leg's controller to its own local data, the decentralized architectures effectively prevented the "brain" from over-relying on complex, body-wide correlations. This acted as a form of regularization, leading to faster learning and the development of robust, modular walking primitives.

2. Superiority in Sparse Reward Scenarios

Flat architectures (Cen and Dec) failed entirely in Task 2 and Task 3, achieving a 0% success rate. Because these models had to learn both walking and navigation simultaneously with rare feedback, they could not converge. In contrast, the hierarchical models—specifically the C_D architecture—achieved a 99.6% success rate on flat terrain by leveraging pre-trained walking skills.

3. Energy Efficiency and Robustness

Decentralized hierarchical models (C_D and D_D) proved significantly more power-efficient than flat models. They produced smoother movements with lower torque requirements, avoiding the jittery, high-energy motions common in standard RL. Furthermore, these models maintained high success rates and efficiency when tested on unseen "bumpy" terrain, demonstrating superior generalization.

4. The "C_D" Sweet Spot

The architecture featuring a Centralized HLC and Decentralized LLC (C_D) emerged as the most effective. It provides a unified point for strategic decision-making (navigation) while maintaining the modularity and efficiency of decentralized motor execution.

Conclusion

This work demonstrates that hierarchical and decentralized architectures are not just biologically inspired curiosities but are functional necessities for solving complex, sparse-reward tasks in legged locomotion. By decoupling the "what to do" (strategy) from the "how to do it" (execution), and further modularizing the execution by limb, agents can achieve higher energy efficiency and better robustness. Future work will focus on transitioning these controllers from simulation to physical hardware and exploring dynamic hierarchies that can adapt to mechanical failures.

Summary updated

Abstract

Legged locomotion is widespread in nature and has inspired the design of current robots. The controller of these legged robots is often realized as one centralized instance. However, in nature, control of movement happens in a hierarchical and decentralized fashion. Introducing these biological design principles into robotic control systems has motivated this work. We tackle the question whether decentralized and hierarchical control is beneficial for legged robots and present a novel decentral, hierarchical architecture to control a simulated legged agent. Three different tasks varying in complexity are designed to benchmark five architectures (centralized, decentralized, hierarchical and two different combinations of hierarchical decentralized architectures). The results demonstrate that decentralizing the different levels of the hierarchical architectures facilitates learning of the agent, ensures more energy efficient movements as well as robustness towards new unseen environments. Furthermore, this comparison sheds light on the importance of modularity in hierarchical architectures to solve complex goal-directed tasks. We provide an open-source code implementation of our architecture (https://github.com/wzaielamri/hddrl).
BibTeX
@inproceedings{amri2022hierarchical,
  author = {Wadhah Zai El Amri and Luca Hermes and Malte Schilling},
  title = {{Hierarchical Decentralized Deep Reinforcement Learning Architecture for a Simulated Four-Legged Agent}},
  booktitle = {LOD},
  year = {2022},
  doi = {10.1007/978-3-031-25891-6\_20},
}
Figure