Abstract
We introduce Glinthawk, an architecture for offline Large Language Model (LLM) inference. By leveraging a two-tiered structure, Glinthawk optimizes the utilization of the high-end accelerators ("Tier 1") by offloading the attention mechanism to lower-end compute tier ("Tier 2"). This separation allows the memory demand of the attention, known as the key-value cache, to scale independently from the model weights, enabling larger batch sizes and more efficient accelerator usage. Prototyped with NVIDIA T4 GPUs and standard CPU VMs, Glinthawk improves throughput by $5.9\times$ and reduces cost of generation by $2.8\times$, compared to paged attention baselines. For long sequence lengths, it achieves $16.3\times$ throughput improvement at $2.4\times$ less cost. Our evaluation shows that this architecture can tolerate moderate network latency with minimal performance degradation, making it highly effective for latency-tolerant, throughput-focused applications such as batch processing. The prototype is publicly available at https://github.com/microsoft/glinthawk.
Introduction: The LLM Memory Wall
In Large Language Model (LLM) serving, we are increasingly hitting a "Memory Wall." High-end GPUs like the NVIDIA H100 excel at computation, but their high-bandwidth memory (HBM) is a scarce resource. For offline tasks—such as processing massive document sets or dataset cleaning—the primary goals are total throughput and cost-efficiency rather than millisecond-level latency. Current serving systems often leave GPUs underutilized because the available memory is consumed by the Key-Value (KV) cache, limiting the batch size.
The Glinthawk Architecture
Glinthawk addresses this by decoupling memory demand from compute power through a "dissected" two-tiered approach.
- Tier 1 (The Muscle): High-performance accelerators (e.g., GPUs) handle "non-attention" operations. These are heavy matrix multiplications (GEMMs) that are stateless and benefit significantly from large batch sizes.
- Tier 2 (The Memory): Low-end, scalable compute nodes (e.g., standard CPU VMs with DRAM) handle the attention mechanism. This involves storing the KV cache and performing memory-heavy but compute-light vector multiplications (GEMVs).
By moving the stateful KV cache to Tier 2, Tier 1 becomes stateless. Communication between tiers is handled via standard Ethernet, using asynchronous processing and multiple "in-flight" batches to hide network latency.
Key Insights
- Parallel Attention: The attention mechanism for different prompts is "embarrassingly parallel," allowing the KV cache to be spread across a swarm of cheap nodes without complex synchronization.
- Latency-Throughput Trade-off: Offline processing can tolerate higher per-token latency in exchange for massive aggregate throughput. Glinthawk exploits this by using asynchronous communication to keep the GPU constantly busy.
Evaluation and Results
The researchers prototyped Glinthawk using NVIDIA T4 GPUs for Tier 1 and AMD EPYC CPUs for Tier 2, evaluating it with Llama 2-70B on the ShareGPT dataset.
Glinthawk achieved a 5.9x throughput increase and a 2.8x cost reduction compared to state-of-the-art paged attention baselines like vLLM. For long-context workloads, the benefits are even more pronounced, reaching a 16.3x throughput improvement. The system also demonstrated high resilience to network latency, maintaining performance even with inter-tier latencies of tens of milliseconds.
Conclusion
Glinthawk demonstrates that rethinking the hardware-software boundary for LLM inference can yield dramatic efficiency gains. While not intended for real-time interactive chat, it provides a powerful solution for high-volume, offline batch processing. Future work includes improving failure recovery for Tier 2 nodes and exploring specialized hardware designs optimized for the Tier 2 memory-heavy workload.
The prototype is available at: https://github.com/microsoft/glinthawk.
Summary updated