TL;DR
HyCoInfer is a heterogeneous cloud–edge serving system that optimizes Large Language Model (LLM) inference by splitting the workload: cloud GPUs handle the compute-heavy "prefill" phase, while edge FPGAs and cloud GPUs share the memory-bound "decoding" phase. By combining adaptive KV cache compression (RVQ) and layer-wise pipelining, it achieves up to 4.11× higher throughput and 61% lower latency than standard cloud-only deployments.
What Problem Does This Paper Address?
Serving LLMs is notoriously expensive and resource-intensive. The process happens in two distinct stages:
- Prefill: The model processes the entire input prompt at once. This is "compute-bound," meaning it saturates the massive parallel processing power of GPUs.
- Decoding: The model generates tokens one by one. This is "memory-bound," meaning the GPU's arithmetic units sit mostly idle while waiting for data to move from memory.
While edge FPGAs are highly efficient at the decoding phase, they lack the raw horsepower for prefill. Conversely, using high-end cloud GPUs for decoding is a waste of their compute capacity. Simply offloading decoding to the edge is difficult because the Key-Value (KV) cache—the "memory" of the conversation—is massive. Transferring this cache over constrained wide-area network (WAN) links often creates a bottleneck that outweighs any gains from edge execution.
Methods
HyCoInfer introduces a collaborative architecture that treats cloud GPUs and edge FPGAs as a unified, heterogeneous resource pool.
Key technical innovations include:
- FPGA-Aware PD Balancing: A scheduling strategy that dynamically partitions GPUs between prefill and decode roles, specifically accounting for the extra decoding "boost" provided by edge FPGAs.
- Heterogeneity-Aware Dispatch: The system routes requests based on prompt length and network state. Short prompts (which have smaller KV caches) are favored for the FPGA, while long prompts stay in the cloud to avoid massive transfer overhead.
- Bandwidth-Adaptive RVQ: It uses Residual Vector Quantization to compress the KV cache. Unlike fixed compression, RVQ is "progressive"—if the network is slow, the system can send fewer "residual stages" to reduce data volume at a slight cost to model quality.
- Layer-wise Pipelined Transfer: Instead of waiting for the entire prefill to finish, HyCoInfer sends KV cache layers to the edge as soon as they are computed, overlapping communication with computation.
- Edge-Prefix Precomputation: If an edge FPGA is idle, it can proactively compute the KV cache for the first part of a prompt locally, further reducing the amount of data that needs to be sent from the cloud.
Datasets
The authors evaluated HyCoInfer using Llama-3.1-8B-Instruct across four diverse serving workloads:
- IMDB: Short-text sentiment classification.
- CNN/DailyMail: Medium-length document summarization.
- Qasper: Long-document question answering (5K–16K tokens).
- GovReport: Long-form summarization.
For model quality benchmarks, they used ARC-Challenge, HellaSwag, MMLU, and WinoGrande to ensure that their KV cache compression didn't degrade the model's reasoning capabilities.
Key Insights and Ideas
- Asymmetry as an Advantage: The core "aha moment" is realizing that the performance asymmetry between GPUs (prefill kings) and FPGAs (decode specialists) can be perfectly balanced if you treat them as a single pipeline rather than separate silos.
- Communication is the Real Boss: In cloud-edge systems, the network is the primary constraint. HyCoInfer’s focus on adaptive compression (RVQ) allows the system to maintain a "goodput" (SLO-compliant throughput) even when network conditions fluctuate.
- Pipelining Everything: By breaking the KV cache into layers and streaming them, the authors effectively hide the "migration" cost that usually kills the performance of disaggregated systems.
Key Results
The system was tested using NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 FPGA over links ranging from 100 to 1000 Mbps.
- Throughput Gains: At 500 Mbps, HyCoInfer delivered 4.11× the request goodput of the popular vLLM framework on the Qasper dataset.
- Latency Reduction: It reduced mean end-to-end latency by up to 61.1% compared to cloud-only baselines.
- Energy Efficiency: Collaborative execution significantly lowered the energy consumed per request, as FPGAs are far more power-efficient for the decoding task than GPUs.
- Robustness: Even at a low 100 Mbps bandwidth, HyCoInfer maintained a 3.45× goodput advantage over unoptimized cloud-edge baselines.
Limitations and Future Work
- Model Scale: The evaluation focused on an 8B parameter model. While the techniques are general, much larger models (e.g., 70B+) would generate even larger KV caches, potentially requiring even more aggressive compression or higher bandwidth.
- Hardware Specificity: The current implementation relies on the FlightLLM kernel for FPGAs. Porting this to other edge accelerators (like specialized AI NPU chips) would require additional engineering effort.
- Dynamic Environments: While the system adapts to bandwidth, future work could explore even more granular scheduling that accounts for fluctuating power costs or mobile edge nodes with intermittent connectivity.
Summary updated