HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment

Youhe Jiang, Ran Yan, Binhang Yuan

Abstract

Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding interference and optimizes resource allocation. However, it is still an open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative to deployment over homogeneous high-performance GPUs. Towards this end, we introduce HexGen-2, a distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HexGen, the core component of HexGen-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.

Introduction

Serving Large Language Models (LLMs) at scale is notoriously expensive, typically requiring clusters of identical, high-end GPUs like the NVIDIA H100. However, many organizations possess a "mishmash" of hardware, including older or mid-tier GPUs such as A100s, L40s, and A6000s. Utilizing this heterogeneous hardware for LLM inference is challenging due to significant differences in compute power, memory bandwidth, and interconnect speeds.

A critical bottleneck in LLM serving is the interference between the two distinct phases of inference: Prefill and Decoding. The prefill phase, which processes the input prompt, is compute-heavy, while the decoding phase, which generates tokens sequentially, is memory-bandwidth heavy. As shown in the throughput analysis below, these phases exhibit vastly different scaling behaviors as batch sizes increase.

Throughput characteristics of the Prefill and Decode phases across different batch sizes.

While "disaggregated inference"—splitting these phases onto different GPUs—is a known solution for uniform clusters, applying it to a diverse mix of hardware introduces complex scheduling challenges. HexGen-2 addresses this by optimizing the allocation of tasks across heterogeneous resources to avoid bottlenecks and maximize hardware utility.

Methodology: The HexGen-2 System

HexGen-2 introduces a novel three-stage scheduling algorithm designed to solve the resource allocation puzzle in heterogeneous environments. The system follows the disaggregated paradigm, where prefill and decoding tasks are handled by separate model replicas, necessitating the transfer of the Key-Value (KV) cache between phases.

The disaggregated inference architecture of HexGen-2, showing the transfer of KV cache between prefill and decode model replicas.

Scheduling Algorithm

  1. Graph Partitioning: The GPU cluster is modeled as a graph where nodes represent GPUs and edges represent network connections. Spectral partitioning is used to group GPUs into "serving groups," ensuring each group has sufficient memory to host a model replica.
  2. Max-Flow Optimization: The problem is transformed into a directed flow network. By applying a max-flow algorithm, HexGen-2 determines the optimal parallel strategies (Tensor and Pipeline Parallelism) for each group and identifies the most efficient paths for inter-phase KV cache communication.
  3. Iterative Refinement: The system employs a "max-flow guided edge swap" technique to iteratively identify bottlenecks and reconfigure GPU groups until a near-optimal setup is achieved.

HexGen-2 is designed to handle complex network topologies, accounting for varying bandwidths from high-speed NVLink to standard PCIe and Ethernet connections.

Experimental Evaluation

The researchers evaluated HexGen-2 using OPT (30B) and Llama-2 (70B) models. Workloads were derived from the Azure Conversation dataset, categorized into types such as Heavy Prefill/Light Decoding (HPLD) and Heavy Prefill/Heavy Decoding (HPHD). Experiments were conducted on real-world heterogeneous clusters featuring mixtures of H100, A100, L40, and A6000 GPUs.

Key Results

  • Throughput: HexGen-2 achieved up to a 2.0x improvement (1.3x average) in serving throughput compared to state-of-the-art systems like DistServe and the original HexGen.
  • Latency: Average inference latency was reduced by 1.5x under the same price budget.
  • Cost Efficiency: HexGen-2 delivered performance comparable to high-end homogeneous clusters while requiring a 30% lower price budget.

Discussion and Insights

A primary innovation of HexGen-2 is its treatment of the KV cache transfer. In disaggregated systems, moving this "memory" of the conversation is often the most expensive operation. HexGen-2 prioritizes high-bandwidth links specifically for these inter-phase transfers.

Furthermore, the system leverages heterogeneity as an advantage. It can assign compute-heavy prefill tasks to GPUs with high TFLOPS and memory-heavy decoding tasks to those with superior HBM bandwidth. The use of max-flow to model inference requests as a fluid moving through pipes naturally reveals hardware "clogs" or bottlenecks, allowing for more efficient resource utilization than traditional uniform approaches.

Conclusion and Future Work

HexGen-2 demonstrates that a "scrapped together" cluster of diverse GPUs can be as effective as the latest flagship hardware when managed by intelligent scheduling software. While the current scheduling algorithm may take significant time to converge for very large clusters (e.g., 50 minutes for 320+ GPUs), this is a one-time cost. Future work could improve the system's robustness to fluctuating network bandwidth in cloud environments and expand support to non-NVIDIA hardware like AMD GPUs or specialized NPUs.

Summary updated

Abstract

Disaggregating the prefill and decoding phases represents an effective new paradigm for generative inference of large language models (LLM), which eliminates prefill-decoding interference and optimizes resource allocation. However, it is still an open problem about how to deploy the disaggregated inference paradigm across a group of heterogeneous GPUs, which can be an economical alternative to deployment over homogeneous high-performance GPUs. Towards this end, we introduce HexGen-2, a distributed system for efficient and economical LLM serving on heterogeneous GPUs following the disaggregated paradigm. Built on top of HexGen, the core component of HexGen-2 is a scheduling algorithm that formalizes the allocation of disaggregated LLM inference computations and communications over heterogeneous GPUs and network connections as a constraint optimization problem. We leverage the graph partitioning and max-flow algorithms to co-optimize resource allocation, parallel strategies for distinct inference phases, and the efficiency of inter-phase key-value (KV) cache communications. We conduct extensive experiments to evaluate HexGen-2, i.e., on OPT (30B) and Llama-2 (70B) models in various real-world settings, the results reveal that HexGen-2 delivers up to a 2.0 times and on average a 1.3 times improvement in serving throughput, reduces the average inference latency by 1.5 times compared with state-of-the-art systems given the same price budget, and achieves comparable inference performance with a 30% lower price budget.
BibTeX
@inproceedings{jiang2025hexgen,
  author = {Youhe Jiang and Ran Yan and Binhang Yuan},
  title = {{HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=Cs6MrbFuMq},
}
Figure