SDE-HARL: Scalable Distributed Policy Execution for Heterogeneous-Agent Reinforcement Learning

Toan Gian, Mohammad Abdi, Nathaniel D. Bastian, Francesco Restuccia

Abstract

Heterogeneous-Agent Reinforcement Learning (HARL) enables agents to execute cooperative tasks by adopting agent-specific policies. Most existing HARL methods use individual policy neural networks to ensure monotonic improvement, which leads to substantial computational overhead. The proposed SDE-HARL overcomes this limitation by decomposing each agent's policy neural network into a lightweight local neural network and a global neural network executed at an edge server. Each local neural network generates and sends a compressed latent representation to the edge server, which aggregates the representations and produces agent-specific inferences. As such, SDE-HARL allows to significantly save computing and networking resources while preserving agent-specific behavior. A key feature of SDE-HARL is grouping agents with similar roles via a role-aware mechanism and sharing partial parameters in their global networks, while an identity-aware mechanism is introduced to promote behavioral diversity among agents within the same group. Experimental results show that SDE-HARL reaches up to 90% win rate while reducing latency, energy consumption, and networking overhead respectively by 2x, 2.5x, and 5x compared to existing work.

Introduction

In Multi-Agent Reinforcement Learning (MARL), "heterogeneous" systems—where agents have different capabilities, sensors, or roles—are more realistic but significantly harder to scale. Most current HARL algorithms follow the Centralized Training, Decentralized Execution (CTDE) paradigm, which faces a "double bind" in real-world deployments. First, running complex, agent-specific policy networks locally on mobile devices like drones or small robots is often impossible due to limited RAM and processing power. Second, offloading tasks to a server by sending raw sensor data creates massive bandwidth congestion that fails to scale with the number of agents. Furthermore, existing HARL theories require agents to have distinct policies to guarantee monotonic improvement, preventing the use of simple parameter-sharing tricks common in MARL.

Methodology

The authors introduce SDE-HARL (Scalable Distributed Policy Execution), which utilizes three main innovations to address these bottlenecks:

Split Policy Execution

Instead of a single monolithic network, the policy is split between the agent and the server. A lightweight Local DNN on the agent extracts features, which are then compressed and sent to a Global DNN on an edge server. The server performs the heavy lifting and returns specific actions for each agent.

The SDE-HARL system architecture showing the split between local agents and the edge server.

Latent Feature Compression

To solve the networking bottleneck, SDE-HARL avoids sending raw data. It uses a "bottleneck" layer and entropy coding to transmit highly compressed latent representations, sending only the information necessary for decision-making. By integrating the compression rate directly into the RL objective function, agents effectively learn to communicate concisely.

Selective Parameter Sharing (RAD & IAD)

To maintain efficiency without sacrificing specialization, SDE-HARL employs two diversity mechanisms:

  • Role-Aware Diversity (RAD): Agents with similar behaviors are clustered into groups that share the same Global DNN backbone, saving memory.
  • Identity-Aware Diversity (IAD): To prevent agents in the same group from becoming identical, an intrinsic reward based on mutual information encourages agents to maintain unique behavioral identities even while sharing a network backbone.
Detailed architecture of SDE-HARL including the role-aware clustering and identity-aware global DNNs.

Experimental Evaluation

The framework was evaluated on three major reinforcement learning benchmarks: Google Research Football (GRF), StarCraft Multi-Agent Challenge (SMAC), and SMACv2. The authors also built a physical testbed using Raspberry Pi 5 units as agents and an NVIDIA RTX 8000 GPU as the edge server to measure real-world latency and energy consumption.

Key Results and Insights

  • Performance and Scalability: SDE-HARL achieved a 15% higher win rate than state-of-the-art baselines in complex scenarios like the "Corridor" task in SMAC. It maintained high performance even as the number of agents scaled up to 27.
  • Efficiency Gains: Compared to standard HARL methods, SDE-HARL reduced latency by 2x, energy consumption by 2.5x, and networking overhead by 5x.
  • Optimal Splitting: The authors found that splitting the network at the 3rd layer (out of 10) provided the best trade-off between local abstraction and server-side processing.

Conclusion

SDE-HARL demonstrates that it is possible to achieve the efficiency of shared knowledge alongside the effectiveness of specialized skills in heterogeneous tasks. While the system significantly reduces networking load, future work is needed to address potential packet loss and explore dynamic role-switching where agents can change their shared backbone as mission requirements evolve.

Summary updated

Abstract

HARL enables agents to execute cooperative tasks by adopting agent-specific policies. Most of existing HARL methods use individual policy neural networks to ensure monotonic improvement, which leads to substantial computational overhead. The proposed SDE-HARL overcomes this limitation by decomposing each agent's policy neural network into a lightweight local neural network and a global neural network executed at an edge server. Each local neural network generates and sends a compressed latent representation to the edge server, which aggregates the representations and produces agent-specific inferences. As such, SDE-HARL allows to significantly save computing and networking resources while preserving agent-specific behavior. A key feature of SDE-HARL is grouping agents with similar roles via a role-aware mechanism and share partial parameters in their global networks, while an identity-aware mechanism is introduced to promote behavioral diversity among agents within the same group. We prototyped SDE-HARL on an experimental testbed composed of a Jetson Nano and Raspberry PI to measure latency and network resource consumption. We evaluated SDE-HARL's performance on several benchmark datasets, including Google Research Football and StarCraft II. Experimental results show that SDE-HARL reaches up to 90% win rate while reducing latency, energy consumption, and networking overhead respectively by 2x, 2.5x, and 5x compared to existing work.
BibTeX
@inproceedings{gian2026sde,
  author = {Toan Gian and Mohammad Abdi and Nathaniel D. Bastian and Francesco Restuccia},
  title = {{SDE-HARL: Scalable Distributed Policy Execution for Heterogeneous-Agent Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i25.39274},
}
Figure