ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference

Hung-Chun Lin, Ting-Wei Hsu, Chung-En Ho, Ahmed SaeedarXiv 2026 · DOI · arXiv

TL;DR

ARK is a lightweight, host-based path reservation system designed to eliminate network congestion in disaggregated LLM inference clusters. By coordinating how "mega elephant flows" (massive KV-cache transfers) are routed through the data center fabric, ARK reduces tail flow completion times by up to 34% and improves user-perceived latency (TTFT) by nearly 13% without requiring any changes to network hardware.

What Problem Does This Paper Address?

Modern Large Language Model (LLM) serving is increasingly moving toward disaggregated inference. In this architecture, the "prefill" phase (processing the input prompt) and the "decode" phase (generating tokens one by one) are handled by separate GPU pools. This allows for better resource scaling, but it introduces a massive new bottleneck: the KV-cache transfer.

When prefill finishes, the resulting KV-cache—which can reach tens of gigabytes for models like LLaMA-3.1-405B—must be shipped across the network to a decode GPU before generation can start. These transfers are "mega elephant flows" that can saturate a 400 Gbps link for nearly a second.

Standard data center networks use ECMP (Equal-Cost Multi-Path) routing, which relies on hashing packet headers to choose a path. Because ECMP is oblivious to flow size, multiple massive KV-cache transfers often "collide" on the same network link (spine), while other links sit idle. These collisions cause queuing delays that directly inflate the Time-to-First-Token (TTFT), a critical user-experience metric.

Methods

The authors propose ARK, a distributed path reservation mechanism that runs entirely in end-host software. ARK exploits two unique characteristics of KV-cache flows: they are long-lived (hundreds of milliseconds), and their size and destination are known the moment the prefill engine finishes.

  • Path Reservation Table: Each host maintains a local, replicated table of which network "spines" are currently occupied by active transfers.
  • Source-Port Manipulation: Instead of letting the OS pick a random source port, the ARK controller selects a port that is mathematically guaranteed to hash the flow onto an unreserved spine under standard ECMP.
  • Lightweight Coordination: When a flow arrives, the host reserves a path, starts the transfer, and broadcasts the reservation to its peers via a low-bandwidth control channel.
  • Race Condition Resolution: If two hosts try to reserve the same path simultaneously, ARK uses a deterministic timestamp-based rule to decide the winner, forcing the loser to re-hash to a different path.
Figure 1: (a) Network topology (b) ARK path reservation controller (c) ARK deployment with separated control and data planes

Datasets

The authors evaluated ARK using a combination of:

  • SimAI/NS-3: A packet-level RDMA network simulator.
  • Vidur: An LLM inference simulator used to generate realistic prefill and decode latencies based on LLaMA-3-8B profiles on NVIDIA H100 GPUs.
  • Workload Traces: Requests were generated using a Poisson arrival process with prompt lengths following a Zipf distribution ($\alpha=0.7$), reflecting the heavy-tailed nature of real-world LLM usage where occasional very long prompts create massive KV-cache flows.

Key Insights and Ideas

  • Coordination is "Free": While per-packet or per-flowlet coordination is usually too slow for data centers, KV-cache flows are so long that the few microseconds needed for ARK's broadcast coordination represent less than 0.01% of the total transfer time.
  • Zero Hardware Changes: Unlike prior research (like CONGA or ConWeave) that requires programmable switches or custom ASICs, ARK works on commodity "dumb" switches by simply being smarter about the 5-tuple headers it sends.
  • Predictability is Power: ARK moves from a reactive posture (responding to congestion after it happens) to a proactive one (avoiding collisions before the first byte is even sent).

Key Results

ARK significantly outperforms vanilla ECMP across several metrics:

  • Hotspot Reduction: As shown in the heatmap below, ECMP often piles up to 18 concurrent flows onto a single spine while others are idle. ARK spreads the load evenly, keeping peaks below ~8 flows.
  • Flow Completion Time (FCT): ARK reduces mean FCT by up to 27.3% and P95 (tail) FCT by up to 34.0% under moderate loads.
  • User Latency: These network gains translate to a 12.9% reduction in mean TTFT, significantly improving the responsiveness of the LLM service.
Figure 3: Per-spine active-flow heatmap at QPS = 25 (top: ECMP, bottom: ARK). ECMP drives persistent hotspots of up to 18 concurrent flows onto Spines 2 and 3 while Spines 1 and 4 stay lightly loaded; ARK keeps peaks below ∼8 across all four spines.
Figure 4: TTFT across the same QPS sweep as Fig. 2.

Limitations and Future Work

  • Clock Synchronization: ARK assumes host clocks are synchronized (e.g., via PTP) to resolve race conditions. While common in modern AI clusters, it is a prerequisite.
  • Control Plane Dependency: For best results, ARK benefits from a separate management channel or low-priority queue for its coordination messages to avoid them getting stuck behind the very elephant flows they are trying to manage.
  • Topology Scope: The evaluation focused on a two-tier spine-leaf topology. While the authors provide a mathematical basis for extending ARK to deeper Clos fabrics, those scenarios were not explicitly simulated in this paper.

Summary updated

Abstract

Disaggregated LLM inference separates the prefill and decode phases across GPU pools, generating massive KV-cache transfers. Because these transfers last hundreds of milliseconds to seconds, they behave as mega elephant flows that dominate link bandwidth utilization. In this regime, stateless ECMP can perform poorly: hash collisions may overload one spine link while leaving others idle, stretching transfer times by seconds. Yet this same persistence makes coordination practical. Since these flows are long-lived, even lightweight one-to-all coordination can be amortized over their lifetime. We present ARK, a distributed elephant-flow path reservation mechanism. ARK coordinates senders to choose source ports whose hashes map concurrent flows onto distinct spines, without requiring switch changes or receiver-side packet reordering. Packet-level RDMA simulations show that ARK reduces mean and P95 FCT by up to 27.3% and 34.0% under moderate load, and further reduces mean TTFT by up to 12.9%.
BibTeX
@misc{lin2026ark,
  author = {Hung-Chun Lin and Ting-Wei Hsu and Chung-En Ho and Ahmed Saeed},
  title = {{ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference}},
  year = {2026},
  eprint = {2610.41160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41160},
}
Figure