SDE-HARL: Scalable Distributed Policy Execution for Heterogeneous-Agent Reinforcement Learning

Toan D. Gian, Mohammad Abdi, Nathaniel D. Bastian, Francesco Restuccia

TL;DR

SDE-HARL is a new framework for Heterogeneous-Agent Reinforcement Learning (HARL) that splits policy execution between resource-constrained local agents and a powerful edge server. By using latent feature compression and a role-aware parameter-sharing mechanism, it achieves high performance (up to 90% win rates) while reducing latency by 2x and networking overhead by 5x compared to traditional methods.

What Problem Does This Paper Address?

In Multi-Agent Reinforcement Learning (MARL), "heterogeneous" systems—where agents have different capabilities, sensors, or roles—are more realistic but much harder to scale. Most current HARL algorithms follow the Centralized Training, Decentralized Execution (CTDE) paradigm. This creates a "double bind" for real-world deployments:

  1. The Compute Bottleneck: Running complex, agent-specific policy networks locally on mobile devices (like drones or small robots) is often impossible due to limited RAM and processing power.
  2. The Networking Bottleneck: Offloading tasks to a server by sending raw sensor data (like high-frequency LiDAR or video) creates massive bandwidth congestion that doesn't scale as you add more agents.

Furthermore, existing HARL theories require agents to have distinct policies to guarantee "monotonic improvement" (the mathematical assurance that the team gets better over time). This prevents researchers from using simple parameter-sharing tricks that usually make MARL more efficient.

Methods

The authors introduce SDE-HARL (Scalable Distributed Policy Execution), which uses three main innovations:

  • Split Policy Execution: Instead of one giant network, the policy is split. A lightweight Local DNN on the agent extracts features, which are then compressed and sent to a Global DNN on an edge server. The server performs the heavy lifting and sends back the specific action for that agent.
  • Latent Feature Compression: To solve the networking bottleneck, SDE-HARL doesn't send raw data. It uses a "bottleneck" layer and entropy coding to transmit highly compressed latent representations. This allows the system to send only the information necessary for decision-making.
  • Selective Parameter Sharing (RAD & IAD):
    • Role-Aware Diversity (RAD): Agents with similar behaviors are clustered into groups that share the same "Global DNN" backbone, saving memory.
  • Identity-Aware Diversity (IAD): To prevent agents in the same group from becoming identical "clones," the authors use an intrinsic reward based on mutual information. This encourages agents to maintain unique behavioral identities even while sharing a network backbone.
  • Two-Phase Training: To keep the mathematical guarantees of HARL, they train the shared backbone first, then freeze it and fine-tune agent-specific "heads" sequentially.

Datasets

The framework was evaluated on three major reinforcement learning benchmarks:

  • Google Research Football (GRF): Simulates complex soccer plays requiring high coordination.
  • StarCraft Multi-Agent Challenge (SMAC): A standard for micro-management in combat.
  • SMACv2: A more difficult version of SMAC with randomized unit compositions and positions, specifically designed to test heterogeneity.

The authors also built a physical testbed using Raspberry Pi 5 units (as agents) and an NVIDIA RTX 8000 GPU (as the edge server) to measure real-world latency and energy.

Key Insights and Ideas

  • The "Sweet Spot" of Splitting: The authors found that the depth of the local network matters. Splitting the network at the 3rd layer (out of 10) provided the best trade-off, offering enough abstraction to compress data effectively without overtaxing the local Raspberry Pi.
  • Sharing vs. Specialization: Naive parameter sharing (giving everyone the same brain) fails in heterogeneous tasks. SDE-HARL proves that you can have the best of both worlds: the efficiency of shared "knowledge" and the effectiveness of specialized "skills" through their RAD and IAD mechanisms.
  • Communication as a Learned Skill: By integrating the compression rate directly into the RL objective function, the agents essentially "learn" how to be concise when talking to the server.

Key Results

  • Performance: SDE-HARL achieved a 15% higher win rate than state-of-the-art baselines in complex scenarios like the "Corridor" task in SMAC.
  • Efficiency Gains: Compared to standard HARL methods, SDE-HARL reduced:
  • Latency by 2x.
  • Energy Consumption by 2.5x.
  • Networking Overhead by 5x.
  • Scalability: The system maintained high performance even as the number of agents scaled up to 27, whereas traditional non-sharing methods saw performance degradation due to the sheer complexity of training so many independent networks.

Limitations and Future Work

While SDE-HARL significantly reduces the networking load, it still relies on a stable connection to an edge server. The paper does not extensively explore how the system would handle "packet loss" or intermittent connectivity, which is common in field robotics.

Additionally, the role-grouping (RAD) is currently performed once before policy training. Future work could involve dynamic role-switching, where agents can change their shared backbone on the fly as the mission requirements evolve (e.g., a scout drone becoming a transport drone).

Summary updated

Abstract

HARL enables agents to execute cooperative tasks by adopting agent-specific policies. Most of existing HARL methods use individual policy neural networks to ensure monotonic improvement, which leads to substantial computational overhead. The proposed SDE-HARL overcomes this limitation by decomposing each agent's policy neural network into a lightweight local neural network and a global neural network executed at an edge server. Each local neural network generates and sends a compressed latent representation to the edge server, which aggregates the representations and produces agent-specific inferences. As such, SDE-HARL allows to significantly save computing and networking resources while preserving agent-specific behavior. A key feature of SDE-HARL is grouping agents with similar roles via a role-aware mechanism and share partial parameters in their global networks, while an identity-aware mechanism is introduced to promote behavioral diversity among agents within the same group. We prototyped SDE-HARL on an experimental testbed composed of a Jetson Nano and Raspberry PI to measure latency and network resource consumption. We evaluated SDE-HARL's performance on several benchmark datasets, including Google Research Football and StarCraft II. Experimental results show that SDE-HARL reaches up to 90% win rate while reducing latency, energy consumption, and networking overhead respectively by 2x, 2.5x, and 5x compared to existing work.
BibTeX
@inproceedings{gian2026sde,
  author = {Toan D. Gian and Mohammad Abdi and Nathaniel D. Bastian and Francesco Restuccia},
  title = {{SDE-HARL: Scalable Distributed Policy Execution for Heterogeneous-Agent Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i25.39274},
}

Loading related content...

Figure