Large Language Model (LLM) inference is fundamentally a tale of two entirely different computational workloads masquerading as a single task. When a user interacts with an LLM, the execution proceeds in two distinct phases: prefill and decode.
In the prefill phase, the model processes the entire input prompt simultaneously to compute the initial key-value (KV) cache. In the decode phase, the model generates tokens autoregressively, one by one, using the previously computed KV cache.
Historically, serving systems treated these two phases uniformly, running them sequentially on the same hardware. However, as models have grown and context windows have expanded, this monolithic approach has become a major bottleneck. Modern LLM infrastructure is shifting toward inference phase decoupling—separating prefill and decode operations to optimize for their radically different hardware demands.
The Hardware Bottleneck Divergence
To understand why decoupling works, we must look at how data moves through a GPU during each phase. The two phases are constrained by completely different physical limits of modern hardware:
- Prefill is Compute-Bound: Because the entire prompt is available upfront, the GPU can parallelize matrix multiplications across all input tokens. This results in high arithmetic intensity (many floating-point operations per byte of data loaded), meaning the execution speed is limited by the GPU's raw tensor processing power.
- Decode is Memory-Bandwidth Bound: During decoding, the model generates exactly one token at a time. To produce a single token, the GPU must load billions of model parameters and the entire historical KV cache from High Bandwidth Memory (HBM) into its SRAM caches. Because the arithmetic intensity is incredibly low, the execution speed is strictly limited by how fast data can be transferred from memory, leaving the GPU's compute cores largely idle.
As detailed by Tarek Abdelzaher et al. in an analysis of data-centric AI bottlenecks, these shifting resource profiles mean that any single, static hardware configuration will be heavily underutilized during at least one phase of inference. Zhenxiao Fu et al. demonstrate in LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences that failing to model these phases separately leads to massive errors in predicting resource and energy consumption, precisely because the underlying hardware signatures are so distinct.
The Cost of Phase Interference
When prefill and decode tasks share the same hardware concurrently, they actively interfere with one another. Imagine a GPU currently processing memory-bound decode iterations for twenty active users. Suddenly, a new request arrives with a massive 4,000-token prompt.
If the inference engine schedules this prefill task immediately, the compute-heavy prompt processing will monopolize the GPU's Streaming Multiprocessors (SMs). As a result, the ongoing decoding tasks are forced to stall.
In Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Scheduling, Kunal Jain et al. quantify this phenomenon, showing that mixing these phases causes severe, non-linear performance degradation. This interference manifests as unpredictable spikes in both Time-to-First-Token (TTFT) for new users and Inter-Token Latency (ITL) for existing users. Lei Gao et al. echo this in DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing, noting that the resource-monopolizing nature of prefill operations makes it incredibly difficult to guarantee strict service-level objectives (SLOs) in multi-tenant environments.
Physical Disaggregation: Splitting the Cluster
The most direct solution to phase interference is physical disaggregation: dedicating separate GPU clusters exclusively to prefill or decode tasks.
[User Request] ---> [Prefill Cluster] (Compute-Optimized)
|
v (Stream KV Cache over Network)
[Decode Cluster] (Memory-Optimized) ---> [Output Tokens]
In a disaggregated architecture, a user's prompt is routed first to a compute-optimized prefill node. Once the prefill node computes the initial KV cache, it transfers this state over a high-speed network to a memory-optimized decode node, which handles the remainder of the token generation.
Foteini Strati et al. implemented this paradigm in DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving, demonstrating that decoupling the state allows each pool of hardware to be optimized independently. This physical separation eliminates pipeline bubbles caused by latency disparities.
Furthermore, disaggregation allows for asymmetric parallelization strategies. As YOUHE JIANG et al. show in HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment, a cluster can run high tensor parallelism on prefill nodes to accelerate matrix math, while using pipeline parallelism or memory-efficient sharding on decode nodes to maximize memory capacity and bandwidth.
This disaggregation principle becomes even more vital when dealing with multimodal models that ingest heavy visual or audio inputs. Gursimran Singh et al. proposed an architecture in Efficiently Serving Large Multimodal Models Using EPD Disaggregation that splits inference into three dedicated stages: Encoding, Prefill, and Decoding. This prevents heavy data-ingestion tasks (like processing a high-resolution video frame) from stalling the downstream token generation of other concurrent requests.
Co-locating via Multiplexing and Chunking
When physical separation onto different machines is impractical—such as on edge devices or smaller deployments—systems can achieve phase separation logically through temporal or spatial multiplexing.
Temporal Chunking
Instead of running a massive prefill all at once, the system can break the prompt into smaller chunks. Amey Agrawal introduced this concept with Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. By chunking the prefill phase, the scheduler can "piggyback" a prefill chunk onto an ongoing decode iteration. Because decoding leaves the GPU's compute cores underutilized, the prefill chunk effectively fills the idle compute capacity without delaying the execution of the decode step.
Spatial Partitioning
Rather than alternating in time, spatial multiplexing splits the GPU's internal resources. In DuetServe, Lei Gao et al. dynamically partition a single GPU's Streaming Multiprocessors at runtime. A subset of the SMs is isolated to handle compute-heavy prefill tasks, while the remaining SMs focus entirely on memory-bound decoding, achieving hardware-level isolation without network overhead.
Phase-Specific Optimization
Once the prefill and decode phases are decoupled, developers can apply highly specialized optimizations to each phase independently, rather than relying on a compromised middle ground.
Progressive and Dynamic Quantization
Because the prefill phase establishes the foundational context of a generation, it is highly sensitive to numerical errors. The decode phase, however, often exhibits higher resilience to precision loss.
To exploit this asymmetry, Hao Mark Chen et al. developed Progressive Mixed-Precision Decoding for Efficient LLM Inference, which maintains high precision (e.g., 16-bit) during the prefill pass and early decoding, then progressively lowers weight precision as token generation continues. Similarly, Fangxin Liu et al. present a dynamic precision switching framework in FlexQuant that targets the memory bandwidth bottleneck of token-by-token generation by dropping to low-bit quantization (like INT4) exclusively during decoding, while preserving high precision for context processing.
Asymmetric Hardware Architectures
The separation of phases has also reached hardware design. Standard GPUs use uniform systolic arrays for all matrix operations. However, specialized accelerators can tailor their dataflows to match the phase characteristics.
In the design framework ADOR, the authors explore combining high-throughput Systolic Arrays (optimized for the compute-bound prefill stage) with specialized Multiply-Accumulate (MAC) Trees designed for low-latency, single-vector decoding operations. Chun-Ting Chen et al. propose a similar hybrid approach in Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference, tailoring data reuse patterns specifically to solve the low-utilization dilemma inherent in uniform edge processors.
By recognizing that an LLM is effectively two different models operating at different times, disaggregated serving transforms the problem of inference optimization from a series of compromises into a pair of cleanly isolated, highly efficient engineering tasks.
Key Foundational Work
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems (2023) — contributed 2 concept elements to this direction.