OLAF: Programmable Data Plane Acceleration for Asynchronous Distributed Reinforcement Learning

Nehal Baganal Krishna, Anam Tahir, Firas Khamis, Mina Tahmasbi Arashloo, Michael Zink, Amr RizkarXiv 2025 · DOI · arXiv

Abstract

Distributed Reinforcement Learning (DRL) often suffers from network congestion, leading to stale model updates that can ruin training convergence. This paper introduces OLAF, a programmable in-network accelerator that opportunistically aggregates and replaces DRL updates on the fly within the switch queue. By keeping updates fresh and reducing redundant traffic, OLAF drastically cuts down model staleness and significantly speeds up training convergence.

Introduction

In large-scale AI data centers, distributed machine learning is the dominant workload. When training Reinforcement Learning (RL) models, an asynchronous approach—where workers independently send updates to a Parameter Server (PS) without waiting for global synchronization—is highly efficient for hardware utilization. However, this lack of coordination creates massive bursts of traffic (incast) at network bottlenecks, like Top-of-Rack (ToR) switches.

When queues fill up, packets are dropped or delayed. In DRL, delayed packets mean the PS receives stale model updates. Staleness is particularly toxic in RL because agents end up exploring environments and generating updates based on outdated policies, which severely degrades the model's ability to converge. Existing in-network aggregation solutions are primarily designed for synchronous training and struggle to scale with the unpredictable, multi-tenant nature of asynchronous DRL.

System Design: The OLAF Accelerator

To solve this, we propose OLAF, a hybrid P4-FPGA data-plane accelerator designed to sit in the network (e.g., at a ToR switch). OLAF introduces several novel mechanisms to manage update freshness and congestion.

Overview of the OLAF system architecture showing the Accelerator Engine performing opportunistic aggregation between worker clusters and parameter servers.

Core Mechanisms

  • Opportunistic In-Queue Aggregation: Instead of a standard FIFO queue, OLAF's queue actively inspects incoming packets. If an incoming update belongs to the same cluster as an update already waiting in the queue, OLAF aggregates them on the fly. This avoids stalling traffic while waiting for specific workers.
  • Smart Update Replacement: If an incoming update comes from the exact same worker as a queued update, OLAF simply replaces the old one. In RL, a newer update from the same worker inherently subsumes the older one because it contains more recent experience.
  • Reverse-Path Signaling & Transmission Control: OLAF piggybacks the current queue status onto the Acknowledgement (ACK) packets traveling back to the workers. Workers use this real-time feedback to adjust their transmission rates, preventing congestion before it happens.
  • Formal Verification of Age-of-Model (AoM): We introduce AoM as a mathematical proxy for staleness. Using a Satisfiability Modulo Theories (SMT) solver, we formally verify that worker-side transmission control parameters guarantee fair AoM across different clusters sharing the same network.

Experimental Evaluation

We evaluated OLAF using the "LunarLander-v3" environment from OpenAI Gym, trained via Distributed Proximal Policy Optimization (DD-PPO). Network simulations in ns-3 emulated up to 100 clusters (300 workers) using real-world distributed RL traces.

Key Results

  • Massive Staleness Reduction: In hardware testbed evaluations (using AMD Alveo U55C cards and an Intel Tofino switch), OLAF reduced the average AoM by 69% (at 40 Gbps capacity) and 78% (at 20 Gbps) compared to a standard FIFO queue.
  • Higher Training Rewards: By preventing stale updates from polluting the global model, OLAF delivered approximately 82% to 86.5% higher average rewards at the end of training compared to a FIFO queue under congested conditions.
Comparison of training rewards over iterations for asynchronous training with and without OLAF aggregation versus synchronous methods.
  • Near-Perfect Fairness: In multi-hop, asymmetric network simulations, OLAF's transmission control achieved an AoM fairness index of 0.99, compared to 0.86 for standard FIFO.

Conclusion

OLAF demonstrates that shifting the focus from traditional network metrics to application-level utility metrics like Age-of-Model (AoM) can significantly improve DRL performance. While current limitations include constraints on model size due to packet fragmentation, future work will explore gradient compression and extending opportunistic queue designs to other domains like live video streaming.

Summary updated

BibTeX
@misc{krishna2025olaf,
  author = {Nehal Baganal Krishna and Anam Tahir and Firas Khamis and Mina Tahmasbi Arashloo and Michael Zink and Amr Rizk},
  title = {{OLAF: Programmable Data Plane Acceleration for Asynchronous Distributed Reinforcement Learning}},
  year = {2025},
  eprint = {2507.05876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05876},
}
Figure