When benchmarking LLM serving engines like vLLM and SGLang, the headlines often scream about massive performance gaps—sometimes claiming one engine is 64x faster than the other. But are we comparing architectures, or just default settings? A new paper from researchers at Thompson Rivers University suggests that much of the "SGLang advantage" isn't due to its fancy radix-tree cache, but rather a simple scheduler setting called chunked prefill.
TL;DR
By conducting a controlled experiment on Kubernetes, the authors found that matching the "chunked prefill" configuration between vLLM and SGLang removes up to 78% of the capacity gap and 30–67% of the latency gap. The remaining performance difference is largely driven by request ordering and queueing dynamics near saturation, rather than the underlying cache architecture.
What Problem Does This Paper Address?
In the world of multi-agent LLM applications, requests often share long prefixes (like system instructions or retrieved documents). Serving engines use KV (Key-Value) caching to avoid recomputing these shared tokens. vLLM uses a block-based approach (PagedAttention), while SGLang uses a more granular radix tree (RadixAttention).
Previous benchmarks showed SGLang significantly outperforming vLLM, a win usually attributed to SGLang’s superior cache architecture. However, these engines also differ in how they schedule "prefill" (processing the initial prompt). SGLang enables chunked prefill by default on many GPUs, splitting long prompts into smaller pieces to avoid stalling other requests. vLLM (in the version studied) did not. The authors set out to determine if the reported performance gap was a result of cache design or simply these mismatched scheduler defaults.
Methods
The researchers built a reproducible testbed on Google Kubernetes Engine (GKE) using NVIDIA L4 GPUs. They developed a seeded multi-agent workload generator that simulates realistic arrival patterns (bursty traffic) and complex agent dependencies (planner $\rightarrow$ workers $\rightarrow$ aggregator).
The core of their method was a two-armed controlled experiment:
- Vendor Defaults: Running both engines with their out-of-the-box settings.
- Matched Chunked Prefill: Forcing vLLM to use the same 2,048-token chunked prefill budget that SGLang uses by default.
They measured Time to First Token (TTFT), Time Per Output Token (TPOT), and sustained throughput, using geometric means and paired t-tests to ensure statistical significance.
Datasets
Instead of a static text dataset, the authors used a synthetic workload generator. It samples tokens uniformly from the model's vocabulary (Llama-3.1-8B-Instruct) to create prompts with exact, controllable prefix sharing ratios ($\sigma$). This allowed them to isolate the effect of caching by ensuring that "shared" prefixes were byte-identical at the token level.
Key Insights and Ideas
- The "Generation Stall" Confound: Without chunked prefill, a long prompt takes over the GPU for a single massive iteration. Any other request currently "decoding" (generating text) must wait, leading to massive latency spikes. As seen in Figure 3, chunking interleaves prefill work with decoding, keeping the GPU busy and the queue moving.
- The Capacity Illusion: Near an engine's saturation point, TTFT doesn't just measure engine speed—it measures the growth of a queue. The authors showed that a tiny 4.4% shortfall in throughput can result in a 12x difference in median TTFT simply because the queue builds up over the duration of the benchmark.
- Cache Architecture vs. Scheduling: Surprisingly, the gap actually narrows as prefix sharing increases. If the radix cache were the primary driver, the gap should widen as sharing becomes more complex. This suggests that scheduling overhead and prefill management are the real bottlenecks in current serving stacks.
- The Value of Sharing: Figure 2 illustrates why we care about this in the first place: on an L4 GPU, moving from 0% sharing to 90% sharing increases the theoretical number of concurrent 4k-token requests from 15 to 139.
Key Results
The results, summarized in Figure 1, are striking:
- Capacity: Matching the chunked prefill setting removed 78% of the capacity gap at the highest tested load.
- Latency: Matching the setting removed 30% to 67% of the latency gap (TTFT) across various conditions.
- The Residual: What survived matching? SGLang retained a 2.4x to 6.4x advantage in tail latency (95th percentile TTFT). The authors hypothesize this is due to SGLang’s "longest prefix match" request ordering, which prioritizes requests that are cheapest to serve from the cache.
Limitations and Future Work
- Version Sensitivity: The study used vLLM 0.6.3 and SGLang 0.4.1. vLLM has since released a "V1" engine that enables chunked prefill by default, effectively validating the authors' findings that the old defaults were suboptimal.
- Unattributed Residuals: While chunking explains most of the gap, the remaining tail-latency advantage for SGLang could be due to several factors (request ordering, attention kernels, or memory management) that were not individually isolated.
- Synthetic Workload: The uniform sampling of tokens avoids "near-duplicate" prompts (e.g., the same text with one character changed), which might behave differently in a radix tree vs. a block-hashed cache.
The authors conclude with six recommendations for the community, most notably: always record the resolved configuration (what the engine actually did, not just the flags you passed) and report throughput alongside latency, as one is meaningless without the other near saturation.
Summary updated