TL;DR
ARK is a lightweight, host-based path reservation system designed to eliminate network congestion in disaggregated LLM inference clusters. By coordinating how "mega elephant flows" (massive KV-cache transfers) are routed through the data center fabric, ARK reduces tail flow completion times by up to 34% and improves user-perceived latency (TTFT) by nearly 13% without requiring any changes to network hardware.
What Problem Does This Paper Address?
Modern Large Language Model (LLM) serving is increasingly moving toward disaggregated inference. In this architecture, the "prefill" phase (processing the input prompt) and the "decode" phase (generating tokens one by one) are handled by separate GPU pools. This allows for better resource scaling, but it introduces a massive new bottleneck: the KV-cache transfer.
When prefill finishes, the resulting KV-cache—which can reach tens of gigabytes for models like LLaMA-3.1-405B—must be shipped across the network to a decode GPU before generation can start. These transfers are "mega elephant flows" that can saturate a 400 Gbps link for nearly a second.
Standard data center networks use ECMP (Equal-Cost Multi-Path) routing, which relies on hashing packet headers to choose a path. Because ECMP is oblivious to flow size, multiple massive KV-cache transfers often "collide" on the same network link (spine), while other links sit idle. These collisions cause queuing delays that directly inflate the Time-to-First-Token (TTFT), a critical user-experience metric.
Methods
The authors propose ARK, a distributed path reservation mechanism that runs entirely in end-host software. ARK exploits two unique characteristics of KV-cache flows: they are long-lived (hundreds of milliseconds), and their size and destination are known the moment the prefill engine finishes.
- Path Reservation Table: Each host maintains a local, replicated table of which network "spines" are currently occupied by active transfers.
- Source-Port Manipulation: Instead of letting the OS pick a random source port, the ARK controller selects a port that is mathematically guaranteed to hash the flow onto an unreserved spine under standard ECMP.
- Lightweight Coordination: When a flow arrives, the host reserves a path, starts the transfer, and broadcasts the reservation to its peers via a low-bandwidth control channel.
- Race Condition Resolution: If two hosts try to reserve the same path simultaneously, ARK uses a deterministic timestamp-based rule to decide the winner, forcing the loser to re-hash to a different path.
Datasets
The authors evaluated ARK using a combination of:
- SimAI/NS-3: A packet-level RDMA network simulator.
- Vidur: An LLM inference simulator used to generate realistic prefill and decode latencies based on LLaMA-3-8B profiles on NVIDIA H100 GPUs.
- Workload Traces: Requests were generated using a Poisson arrival process with prompt lengths following a Zipf distribution ($\alpha=0.7$), reflecting the heavy-tailed nature of real-world LLM usage where occasional very long prompts create massive KV-cache flows.
Key Insights and Ideas
- Coordination is "Free": While per-packet or per-flowlet coordination is usually too slow for data centers, KV-cache flows are so long that the few microseconds needed for ARK's broadcast coordination represent less than 0.01% of the total transfer time.
- Zero Hardware Changes: Unlike prior research (like CONGA or ConWeave) that requires programmable switches or custom ASICs, ARK works on commodity "dumb" switches by simply being smarter about the 5-tuple headers it sends.
- Predictability is Power: ARK moves from a reactive posture (responding to congestion after it happens) to a proactive one (avoiding collisions before the first byte is even sent).
Key Results
ARK significantly outperforms vanilla ECMP across several metrics:
- Hotspot Reduction: As shown in the heatmap below, ECMP often piles up to 18 concurrent flows onto a single spine while others are idle. ARK spreads the load evenly, keeping peaks below ~8 flows.
- Flow Completion Time (FCT): ARK reduces mean FCT by up to 27.3% and P95 (tail) FCT by up to 34.0% under moderate loads.
- User Latency: These network gains translate to a 12.9% reduction in mean TTFT, significantly improving the responsiveness of the LLM service.
Limitations and Future Work
- Clock Synchronization: ARK assumes host clocks are synchronized (e.g., via PTP) to resolve race conditions. While common in modern AI clusters, it is a prerequisite.
- Control Plane Dependency: For best results, ARK benefits from a separate management channel or low-priority queue for its coordination messages to avoid them getting stuck behind the very elephant flows they are trying to manage.
- Topology Scope: The evaluation focused on a two-tier spine-leaf topology. While the authors provide a mathematical basis for extending ARK to deeper Clos fabrics, those scenarios were not explicitly simulated in this paper.
Summary updated