Introduction
Large Language Model (LLM) inference is a cornerstone of modern AI, yet it remains notoriously expensive and computationally complex. This complexity stems from a unique two-phase execution pattern: the Prefill phase, which processes the input prompt in parallel, and the Decode phase, which generates responses sequentially, token-by-token. While it is a common industry "rule of thumb" that Prefill is compute-bound and Decode is memory-bound, the underlying microarchitectural reasons have remained fragmented in literature. This paper provides a bottom-up characterization, linking macroscopic performance metrics to low-level hardware root causes.
The Two-Phase Execution Model
To understand LLM performance, one must first examine the structural differences between phases. During Prefill, the model processes all prompt tokens simultaneously to populate the Key-Value (KV) cache. In the subsequent Decode phase, the model generates one token at a time, attending to the previously cached context.
As shown in Figure 1, the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) layers operate differently across these phases. Prefill utilizes high arithmetic intensity, saturating GPU Tensor Cores. Conversely, Decode is stalled by memory dependencies due to exhaustive KV-cache scanning and low data reuse, making it fundamentally memory-bound.
Performance and Resource Utilization
The researchers established a four-dimensional framework to analyze these phases across various workloads (e.g., ShareGPT, long-context summarization). A key finding is the absolute nature of the Prefill/Decode dichotomy.
As illustrated in Figure 6(a), Prefill consistently achieves high Streaming Multiprocessor (SM) throughput, while Decode is limited by DRAM bandwidth. Figure 6(b) demonstrates how latency scales: while FFN layers dominate at typical context lengths, Attention becomes the primary bottleneck as context grows beyond 16K tokens due to its quadratic (Prefill) and linear (Decode) scaling properties.
Energy Predictability
One of the most significant insights is that energy consumption is almost entirely driven by the Decode phase. The study found a near-perfect linear relationship between energy use and the number of output tokens ($R^2 = 0.999$). This allows system administrators to accurately predict the energy cost of a request before it even executes.
Emerging Paradigms: MoE and RAG
The study also redefined the performance boundaries for newer architectures:
- Mixture-of-Experts (MoE): While MoE models (like Qwen3) offer significant speedups by decoupling parameter count from compute cost, they introduce a massive routing overhead during Decode (up to 28% of total time). This routing splits batches into tiny "per-expert" fragments, ruining GEMM efficiency.
- Retrieval-Augmented Generation (RAG): In RAG workflows, the bottleneck often shifts from the GPU to the CPU. As knowledge bases grow, the system becomes constrained by CPU memory bandwidth during the massive similarity matching required over vector stores.
Conclusion and Actionable Guidelines
By systematically connecting high-level symptoms to hardware realities, the authors propose several optimizations for LLM deployment:
- Phase-Decoupled Serving: Route Prefill tasks to compute-heavy GPU instances and Decode tasks to bandwidth-rich instances to maximize resource efficiency.
- Predictability-Driven Scheduling: Leverage the linear predictability of energy and latency to make smarter scheduling and thermal throttling decisions.
- Hardware Co-design: Develop near-memory accelerators specifically for the similarity matching phase in RAG to alleviate CPU bottlenecks.
Summary updated