Abstract
Edge computing addresses the growing data demands of connected-device networks by placing computational resources closer to end users through decentralized infrastructures. This decentralization challenges traditional, fully centralized orchestration, which suffers from latency and resource bottlenecks. We present FAuNO -- Federated Asynchronous Network Orchestrator -- a buffered, asynchronous federated reinforcement-learning (FRL) framework for decentralized task offloading in edge systems. FAuNO adopts an actor-critic architecture in which local actors learn node-specific dynamics and peer interactions, while a federated critic aggregates experience across agents to encourage efficient cooperation and improve overall system performance. Experiments in the PeersimGym environment show that FAuNO consistently matches or exceeds heuristic and federated multi-agent RL baselines in reducing task loss and latency, underscoring its adaptability to dynamic edge-computing scenarios.
Introduction
Edge computing moves computational resources closer to the user to reduce latency, but managing these resources in a massive, decentralized network of IoT devices and edge servers is a significant challenge. Traditional orchestration—deciding which node should handle which task—usually relies on a central controller. However, in large-scale deployments, a central controller becomes a latency bottleneck and a single point of failure.
Reinforcement Learning (RL) is a promising candidate for decentralized decision-making, but it faces two major hurdles in edge environments: device heterogeneity and the "straggler problem." In standard Federated Learning (FL), the entire network must wait for the slowest device to finish its update, leading to stale policies in dynamic environments.
The FAuNO Framework
FAuNO (Federated Asynchronous Network Orchestrator) addresses these challenges by combining Multi-Agent Reinforcement Learning (MARL) with a semi-asynchronous federated approach.
The framework utilizes a Proximal Policy Optimization (PPO) actor-critic architecture with a strategic split:
- Local Actors: Actors are kept local to each node, allowing them to specialize in their specific neighborhood's dynamics and learn the "quirks" of local neighbors.
- Federated Critic: Only the critic is federated. The global critic aggregates experiences from all agents to provide a "big picture" evaluation of system-wide cooperation.
- Buffered Semi-Asynchronous Aggregation: Borrowing from the FedBuff algorithm, FAuNO uses a buffer at a Global Manager. Updates are applied once a threshold of $K$ updates is received, ensuring that faster nodes can contribute multiple times without being stalled by stragglers.
As illustrated in the comparison below, FAuNO is unique in its integration of buffered asynchronous updates and support for open-source simulation environments compared to existing literature.
Experimental Evaluation
The researchers evaluated FAuNO using a combination of realistic simulation tools and real-world data:
- PeersimGym & Ether Tool: Used to generate realistic hierarchical star topologies and test offloading in large-scale networks.
- Alibaba Cluster Trace: Workloads were derived from real-world data center traces, rescaled to fit edge device constraints (e.g., Raspberry Pis).
- Baselines: FAuNO was benchmarked against Least Queues (LQ) and SCOF (a synchronous federated RL algorithm).
Key Results and Insights
- Task Completion: In structured topologies, FAuNO achieved a Finished Task Ratio of approximately 97%, consistently outperforming both LQ and SCOF, particularly as task arrival rates increased.
- Latency Balance: While synchronous baselines like SCOF sometimes showed lower response times, they did so by failing to complete a higher volume of tasks. FAuNO maintained a superior balance, completing more tasks while keeping response times significantly lower than the LQ heuristic.
- Robustness: FAuNO demonstrated remarkable resilience. Even when 80% of network packets were dropped, the system exhibited graceful degradation, maintaining performance levels comparable to local MARL rather than collapsing.
- Non-IID Data Handling: In heterogeneous workloads, the federated critic successfully converged to a stable model, whereas pure MARL agents without federation suffered from inconsistent policies.
Conclusion and Future Work
FAuNO demonstrates that federating only the critic while allowing local actor specialization provides a robust, scalable solution for edge orchestration. While the current results are simulation-based, they provide a strong foundation for future work, which will focus on security against Byzantine nodes, energy constraints for mobile devices, and deployment on physical edge testbeds.
Summary updated