TL;DR
FAuNO (Federated Asynchronous Network Orchestrator) is a semi-asynchronous federated reinforcement learning framework designed for decentralized task offloading across heterogeneous edge computing systems. By federating only the critic via a buffered, non-blocking aggregation mechanism while keeping actor policies local, FAuNO enables continuous training despite stragglers and significantly reduces task drops and latency compared to standard federated and heuristic baselines.
What Problem Does This Paper Address?
In edge and fog computing, resource-constrained devices (like IoT sensors or single-board computers) need to offload tasks to nearby peers or servers to minimize latency and meet strict deadlines. Fully centralized orchestration creates single points of failure and network bottlenecks. While multi-agent reinforcement learning (MARL) provides a path to decentralized decision-making, deploying it across real-world edge devices introduces major hurdles:
- Heterogeneity and Stragglers: Synchronous federated learning (FL) algorithms stall when slow devices (stragglers) delay global aggregation rounds.
- Partial Observability and Non-IID Workloads: Edge nodes only observe their local neighborhood, and different regions experience wildly different traffic patterns, causing purely independent agents to diverge into selfish, non-cooperative behaviors.
Methods
FAuNO formulates decentralized task offloading as a Partially Observable Markov Game (POMG) and combines Actor-Critic RL (PPO with GAE) with semi-asynchronous federated learning:
- Federate the Critic, Specialize the Actor: Each node maintains a local actor network tuned to its specific hardware and immediate neighbors. Only the critic networks—which predict system-wide value—are shared and aggregated globally. This preserves localized decision-making while aligning agents toward a shared global objective.
- Buffered Semi-Asynchronous Aggregation (Adapted FedBuff): Rather than waiting for every node to finish, a Global Manager (GM) maintains a buffer of critic updates. Slower agents do not block faster agents, and newly arriving updates from an agent overwrite older ones. Once $K$ distinct agent updates accumulate, the GM computes a weighted average (weighted by each agent's local steps) and redistributes the global critic.
- Padded, Normalized State Representation: Observations are standardized with normalized placeholders (e.g.,
-1for absent neighbors), ensuring consistent critic input dimensions despite topological variation. - Network-Aware FL Simulation: The authors extended the
PeersimGymsimulator with anFLManagermodule that accurately models the network overhead, packet size, and transmission delays of exchanging model parameters over edge links.
+-----------------------------------------------------------+
| Local Agent (Node i) |
| - Actor: Local PPO policy (neighborhood actions) |
| - Critic: Periodically pushes updates to Global Manager |
+-----------------------------+-----------------------------+
| Asynchronous updates (Dw)
v
+---------------------------------+
| Global Manager (Buffer size K) |
| - Replaces stale agent updates |
| - Weighted average when k >= K |
| - Broadcasts new Global Critic |
+---------------------------------+
Datasets
The evaluation relies on simulated edge infrastructures combining realistic traces and topologies:
- Workloads: Real-world traces from the Alibaba Cluster Trace dataset, rescaled (10% instruction reduction) to reflect edge device capacities without causing immediate overflow.
- Topologies: Synthesized realistic edge topologies generated via Ether (hierarchical star/AoT clusters comprising Raspberry Pi 3/4/5 nodes, Intel NUC base stations, and edge servers) alongside synthetic 10-node and 15-node random graphs.
Key Insights and Ideas
- Local Actors Prevent Catastrophic Policy Averaging: Averaging actor weights directly across heterogeneous devices severely degrades performance because optimal actions depend on localized link speeds and queues. Federating only the value function (critic) guides cooperative behavior without washing out individual node specialization.
- Graceful Degradation Under Stragglers: In ablation experiments dropping up to 80% of client gradient updates ($s = 0.8$), FAuNO maintained virtually identical finished-task ratios (~0.76) and response times (~260 ticks), proving it gracefully degrades rather than collapses under extreme network unreliability.
- Critic Agreement Under Non-IID Skew: Across heterogeneous clusters processing distinct workloads, FAuNO’s global critic achieved a global disagreement score approaching that of an oracle centralized critic (74.5 vs. 63.6), whereas independent MARL diverged heavily (disagreement scores > 147–313).
Key Results
- Ether Hierarchical Topologies: FAuNO consistently achieved the highest task completion rates across varied arrival rates ($\lambda \in \{0.5, 1, 2\}$), reaching 92.0%–97.4% completion, compared to synchronous FRL baseline SCOF (66.6%–94.3%) and Least Queues (90.9%–94.8%).
- Latency vs. Throughput Trade-off: While SCOF recorded low response times at heavy loads, it did so by failing and dropping over 30% of incoming tasks. FAuNO delivered balanced response times (170–262 ticks) while preserving task delivery.
- Architectural Ablations: Critic-only federation consistently outperformed variants that federated the actor or both actor and critic (e.g., at $\lambda=2$, FAuNO achieved a 0.926 completion ratio vs. 0.736 for federated-actor and 0.848 for actor-critic).
Limitations and Future Work
- Centralized Orchestrator Bottleneck: The framework relies on a single Global Manager to buffer and aggregate updates, introducing a potential single point of failure and scalability limits for large-scale deployments.
- Cooperative & Honest Node Assumption: The current formulation does not guard against adversarial or Byzantine nodes, nor does it incorporate differential privacy.
- Simulation-Only: The framework has been validated solely within simulation (PeersimGym) under static network links; mobile nodes, dynamic wireless fading, and energy consumption tradeoffs remain open topics for future physical-testbed deployment.
Summary updated