Abstract
We present a framework combining hierarchical and multi-agent deep reinforcement learning approaches to solve coordination problems among a multitude of agents using a semi-decentralized model. The framework extends the multi-agent learning setup by introducing a meta-controller that guides the communication between agent pairs, enabling agents to focus on communicating with only one other agent at any step. This hierarchical decomposition of the task allows for efficient exploration to learn policies that identify globally optimal solutions even as the number of collaborating agents increases. We show promising initial experimental results on a simulated distributed scheduling problem.
Introduction
Coordinating multiple autonomous agents is a fundamental challenge in artificial intelligence. Standard Multi-Agent Reinforcement Learning (MARL) often struggles with scalability; as the number of agents grows, the communication overhead and the joint action space expand quadratically. This "noise" from simultaneous multi-agent interactions often prevents the system from learning a coherent global strategy.
While Hierarchical Reinforcement Learning (HRL) excels at decomposing long-term goals into manageable subtasks, it typically assumes a single agent acting sequentially. It does not inherently address scenarios where multiple agents with private information—such as independent web services or autonomous vehicles—must negotiate concurrently to reach a global solution.
The FCRL Framework
To bridge this gap, we introduce Federated Control with Reinforcement Learning (FCRL). This framework utilizes a two-stage hierarchy to manage complex coordination:
- The Meta-Controller: Operating on a slower time scale, this module observes the global state to select a specific subtask (a pair of agents to coordinate) and a constraint (the boundaries for their negotiation, such as a specific time window).
- The Controllers (Agents): These decentralized agents engage in multi-round negotiations when paired. They operate under partial observability, seeing only their private data and the communication from their partner. They are intrinsically motivated to find joint actions that satisfy both the meta-controller's constraints and their internal requirements.
By decomposing a large coordination problem into a sequence of pairwise negotiations, FCRL significantly reduces the complexity of the learning task. The meta-controller abstracts the negotiation details, focusing on high-level sequencing and boundary setting.
Experimental Evaluation
We evaluated the FCRL framework using a Simulated Distributed Scheduling Problem. In this task, $N$ agents (up to 20) each maintain a private database of available time slots. The goal is for a subset of $m$ agents to agree on a schedule where their chosen times follow a specific relative order (e.g., Agent A < Agent B < Agent C).
We tested three levels of complexity: $m=2$ (easy), $m=4$ (medium), and $m=6$ (hard), with a total of 8 possible time slots ($B=8$). FCRL was compared against standard MARL and a basic HRL implementation.
Key Results
The experimental results highlight the scalability of the federated approach:
- Scalability: While all methods performed well in the simple 2-agent setup, the baselines degraded rapidly as $m$ increased.
- Performance Gap: At $m=4$, FCRL significantly outperformed both MARL (which struggled with the search space) and HRL (which lacked necessary inter-agent conflict resolution).
- Robustness: In the hardest scenario ($m=6$), both MARL and HRL failed to achieve any positive rewards. In contrast, FCRL continued to learn and make progress toward an optimal policy.
Discussion and Future Work
FCRL demonstrates that pairwise communication and meta-level constraints are powerful tools for narrowing the search space in multi-agent environments. Furthermore, the framework naturally preserves privacy, as agents only share specific communication vectors with their partners within the bounds set by the meta-controller.
Future research will focus on scaling the action space beyond small databases and moving from heuristic sequencing in the meta-controller to fully learned sequencing. Additionally, transitioning from DQN to policy gradient methods may further stabilize training in these inherently non-stationary multi-agent environments.
Summary updated