HyCoInfer: Cloud-Edge Collaborative Inference for Large Language Models on Heterogeneous GPU-FPGA Architectures

Zewen Huang, Xiangwen Zhuge, Yingkun Hu, Zheng Yang, Qiang MaarXiv 2026 · arXiv

TL;DR

HyCoInfer is a heterogeneous cloud–edge serving system that optimizes Large Language Model (LLM) inference by splitting the workload: cloud GPUs handle the compute-heavy "prefill" phase, while edge FPGAs and cloud GPUs share the memory-bound "decoding" phase. By combining adaptive KV cache compression (RVQ) and layer-wise pipelining, it achieves up to 4.11× higher throughput and 61% lower latency than standard cloud-only deployments.

What Problem Does This Paper Address?

Serving LLMs is notoriously expensive and resource-intensive. The process happens in two distinct stages:

  1. Prefill: The model processes the entire input prompt at once. This is "compute-bound," meaning it saturates the massive parallel processing power of GPUs.
  2. Decoding: The model generates tokens one by one. This is "memory-bound," meaning the GPU's arithmetic units sit mostly idle while waiting for data to move from memory.

While edge FPGAs are highly efficient at the decoding phase, they lack the raw horsepower for prefill. Conversely, using high-end cloud GPUs for decoding is a waste of their compute capacity. Simply offloading decoding to the edge is difficult because the Key-Value (KV) cache—the "memory" of the conversation—is massive. Transferring this cache over constrained wide-area network (WAN) links often creates a bottleneck that outweighs any gains from edge execution.

Methods

HyCoInfer introduces a collaborative architecture that treats cloud GPUs and edge FPGAs as a unified, heterogeneous resource pool.

Fig. 1. HyCoInfer architecture. Cloud GPUs perform prefill and share decoding with edge FPGAs; only compressed KV caches assigned to edge decoders cross the cloud–edge link.

Key technical innovations include:

  • FPGA-Aware PD Balancing: A scheduling strategy that dynamically partitions GPUs between prefill and decode roles, specifically accounting for the extra decoding "boost" provided by edge FPGAs.
  • Heterogeneity-Aware Dispatch: The system routes requests based on prompt length and network state. Short prompts (which have smaller KV caches) are favored for the FPGA, while long prompts stay in the cloud to avoid massive transfer overhead.
  • Bandwidth-Adaptive RVQ: It uses Residual Vector Quantization to compress the KV cache. Unlike fixed compression, RVQ is "progressive"—if the network is slow, the system can send fewer "residual stages" to reduce data volume at a slight cost to model quality.
  • Layer-wise Pipelined Transfer: Instead of waiting for the entire prefill to finish, HyCoInfer sends KV cache layers to the edge as soon as they are computed, overlapping communication with computation.
Fig. 2. Layer-wise pipelining overlaps each KV layer’s RVQ compression, device copy, and cloud–edge transfer with subsequent prefill layers.
  • Edge-Prefix Precomputation: If an edge FPGA is idle, it can proactively compute the KV cache for the first part of a prompt locally, further reducing the amount of data that needs to be sent from the cloud.
Fig. 3. Opportunistic decoder-side precomputation under low edge load. The FPGA computes a prompt prefix locally while receiving cloud-generated suffix KV.

Datasets

The authors evaluated HyCoInfer using Llama-3.1-8B-Instruct across four diverse serving workloads:

  • IMDB: Short-text sentiment classification.
  • CNN/DailyMail: Medium-length document summarization.
  • Qasper: Long-document question answering (5K–16K tokens).
  • GovReport: Long-form summarization.

For model quality benchmarks, they used ARC-Challenge, HellaSwag, MMLU, and WinoGrande to ensure that their KV cache compression didn't degrade the model's reasoning capabilities.

Key Insights and Ideas

  • Asymmetry as an Advantage: The core "aha moment" is realizing that the performance asymmetry between GPUs (prefill kings) and FPGAs (decode specialists) can be perfectly balanced if you treat them as a single pipeline rather than separate silos.
  • Communication is the Real Boss: In cloud-edge systems, the network is the primary constraint. HyCoInfer’s focus on adaptive compression (RVQ) allows the system to maintain a "goodput" (SLO-compliant throughput) even when network conditions fluctuate.
  • Pipelining Everything: By breaking the KV cache into layers and streaming them, the authors effectively hide the "migration" cost that usually kills the performance of disaggregated systems.

Key Results

The system was tested using NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 FPGA over links ranging from 100 to 1000 Mbps.

  • Throughput Gains: At 500 Mbps, HyCoInfer delivered 4.11× the request goodput of the popular vLLM framework on the Qasper dataset.
  • Latency Reduction: It reduced mean end-to-end latency by up to 61.1% compared to cloud-only baselines.
  • Energy Efficiency: Collaborative execution significantly lowered the energy consumed per request, as FPGAs are far more power-efficient for the decoding task than GPUs.
  • Robustness: Even at a low 100 Mbps bandwidth, HyCoInfer maintained a 3.45× goodput advantage over unoptimized cloud-edge baselines.

Limitations and Future Work

  • Model Scale: The evaluation focused on an 8B parameter model. While the techniques are general, much larger models (e.g., 70B+) would generate even larger KV caches, potentially requiring even more aggressive compression or higher bandwidth.
  • Hardware Specificity: The current implementation relies on the FlightLLM kernel for FPGAs. Porting this to other edge accelerators (like specialized AI NPU chips) would require additional engineering effort.
  • Dynamic Environments: While the system adapts to bandwidth, future work could explore even more granular scheduling that accounts for fluctuating power costs or mobile edge nodes with intermittent connectivity.

Summary updated

Abstract

Large language model (LLM) serving relies predominantly on cloud GPUs, yet autoregressive decoding is memory-bound and often leaves their arithmetic units underutilized. Edge FPGAs offer attractive decoding efficiency, but are ill suited to sustaining a high-concurrency serving workload alone. This paper presents HyCoInfer, a heterogeneous cloud-edge serving system that assigns prefill to cloud GPUs and distributes decoding across cloud GPUs and edge FPGAs. The design addresses three barriers to practical offloading: capacity balancing across heterogeneous accelerators, the large KV cache crossing a constrained wide-area link, and the latency added by heterogeneous execution. HyCoInfer incorporates FPGA decode capacity into GPU prefill/decode role partitioning and places requests according to queue load, prompt length, and network state. It further combines layer-wise pipelined transfer, decoder-side precomputation, and bandwidth-adaptive KV cache compression based on residual vector quantization. We implement HyCoInfer using two NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 and evaluate it over 100-1000 Mbps links. At 500 Mbps, HyCoInfer delivers up to 4.11 times the request goodput of cloud-only vLLM and reduces mean end-to-end latency by up to 61.1%. Across different network conditions and workloads, the system improves overall serving capacity, with the largest gains on communication-intensive tasks, and reduces per-request energy.
BibTeX
@misc{huang2026hycoinfer,
  author = {Zewen Huang and Xiangwen Zhuge and Yingkun Hu and Zheng Yang and Qiang Ma},
  title = {{HyCoInfer: Cloud-Edge Collaborative Inference for Large Language Models on Heterogeneous GPU-FPGA Architectures}},
  year = {2026},
  eprint = {2611.22086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22086},
}
Figure