A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

Muhammad Faizan Anwar, Sina KeshvadiarXiv 2026 · arXiv

When benchmarking LLM serving engines like vLLM and SGLang, the headlines often scream about massive performance gaps—sometimes claiming one engine is 64x faster than the other. But are we comparing architectures, or just default settings? A new paper from researchers at Thompson Rivers University suggests that much of the "SGLang advantage" isn't due to its fancy radix-tree cache, but rather a simple scheduler setting called chunked prefill.

TL;DR

By conducting a controlled experiment on Kubernetes, the authors found that matching the "chunked prefill" configuration between vLLM and SGLang removes up to 78% of the capacity gap and 30–67% of the latency gap. The remaining performance difference is largely driven by request ordering and queueing dynamics near saturation, rather than the underlying cache architecture.

What Problem Does This Paper Address?

In the world of multi-agent LLM applications, requests often share long prefixes (like system instructions or retrieved documents). Serving engines use KV (Key-Value) caching to avoid recomputing these shared tokens. vLLM uses a block-based approach (PagedAttention), while SGLang uses a more granular radix tree (RadixAttention).

Previous benchmarks showed SGLang significantly outperforming vLLM, a win usually attributed to SGLang’s superior cache architecture. However, these engines also differ in how they schedule "prefill" (processing the initial prompt). SGLang enables chunked prefill by default on many GPUs, splitting long prompts into smaller pieces to avoid stalling other requests. vLLM (in the version studied) did not. The authors set out to determine if the reported performance gap was a result of cache design or simply these mismatched scheduler defaults.

Methods

The researchers built a reproducible testbed on Google Kubernetes Engine (GKE) using NVIDIA L4 GPUs. They developed a seeded multi-agent workload generator that simulates realistic arrival patterns (bursty traffic) and complex agent dependencies (planner $\rightarrow$ workers $\rightarrow$ aggregator).

The core of their method was a two-armed controlled experiment:

  1. Vendor Defaults: Running both engines with their out-of-the-box settings.
  2. Matched Chunked Prefill: Forcing vLLM to use the same 2,048-token chunked prefill budget that SGLang uses by default.

They measured Time to First Token (TTFT), Time Per Output Token (TPOT), and sustained throughput, using geometric means and paired t-tests to ensure statistical significance.

Datasets

Instead of a static text dataset, the authors used a synthetic workload generator. It samples tokens uniformly from the model's vocabulary (Llama-3.1-8B-Instruct) to create prompts with exact, controllable prefix sharing ratios ($\sigma$). This allowed them to isolate the effect of caching by ensuring that "shared" prefixes were byte-identical at the token level.

Key Insights and Ideas

  • The "Generation Stall" Confound: Without chunked prefill, a long prompt takes over the GPU for a single massive iteration. Any other request currently "decoding" (generating text) must wait, leading to massive latency spikes. As seen in Figure 3, chunking interleaves prefill work with decoding, keeping the GPU busy and the queue moving.
Prefill scheduling: (a) run-to-completion causes stalls, (b) chunked prefill interleaves work.
  • The Capacity Illusion: Near an engine's saturation point, TTFT doesn't just measure engine speed—it measures the growth of a queue. The authors showed that a tiny 4.4% shortfall in throughput can result in a 12x difference in median TTFT simply because the queue builds up over the duration of the benchmark.
  • Cache Architecture vs. Scheduling: Surprisingly, the gap actually narrows as prefix sharing increases. If the radix cache were the primary driver, the gap should widen as sharing becomes more complex. This suggests that scheduling overhead and prefill management are the real bottlenecks in current serving stacks.
  • The Value of Sharing: Figure 2 illustrates why we care about this in the first place: on an L4 GPU, moving from 0% sharing to 90% sharing increases the theoretical number of concurrent 4k-token requests from 15 to 139.
Upper bound on concurrent requests as a function of prefix sharing.

Key Results

The results, summarized in Figure 1, are striking:

  • Capacity: Matching the chunked prefill setting removed 78% of the capacity gap at the highest tested load.
  • Latency: Matching the setting removed 30% to 67% of the latency gap (TTFT) across various conditions.
  • The Residual: What survived matching? SGLang retained a 2.4x to 6.4x advantage in tail latency (95th percentile TTFT). The authors hypothesize this is due to SGLang’s "longest prefix match" request ordering, which prioritizes requests that are cheapest to serve from the cache.
Fig. 1. One setting accounts for most of the gap. Each row shows the engine gap under vendor defaults (filled) and with chunked prefill matched on both engines (hollow), for every condition measured in both configurations. Latency gaps are vLLM’s TTFT divided by SGLang’s; the capacity gap is SGLang’s sustained throughput divided by vLLM’s. The right column gives the fraction of each default gap, on a log scale, that matching removes. Geometric means of three seed-paired runs; ∗marks a paired 𝑡-test on log values with 𝑝< 0.05.

Limitations and Future Work

  • Version Sensitivity: The study used vLLM 0.6.3 and SGLang 0.4.1. vLLM has since released a "V1" engine that enables chunked prefill by default, effectively validating the authors' findings that the old defaults were suboptimal.
  • Unattributed Residuals: While chunking explains most of the gap, the remaining tail-latency advantage for SGLang could be due to several factors (request ordering, attention kernels, or memory management) that were not individually isolated.
  • Synthetic Workload: The uniform sampling of tokens avoids "near-duplicate" prompts (e.g., the same text with one character changed), which might behave differently in a radix tree vs. a block-hashed cache.

The authors conclude with six recommendations for the community, most notably: always record the resolved configuration (what the engine actually did, not just the flags you passed) and report throughput alongside latency, as one is meaningless without the other near saturation.

Summary updated

Abstract

Multi-agent LLM applications repeat most of their prompt tokens across requests, and serving engines reuse the cached state of shared prefixes. vLLM and SGLang reuse it at different granularities, and benchmarks of their default deployments report large performance gaps, usually credited to the cache. Such comparisons rarely control for other configuration differences. One of them, whether a long prompt is prefilled in chunks, is set by each engine's defaults from the hardware and the release, and it confounds the comparison. We built a reproducible testbed on Kubernetes and a seeded multi-agent workload generator, and compared the two engines under vendor defaults and with chunked prefill matched. SGLang's configuration is the same in both arms, which makes it an internal control across independently provisioned deployments. Under defaults, vLLM's median time to first token is up to 64x SGLang's. Matching one setting removes 78% of the capacity gap where that gap is largest and 30 to 67% of the latency gap. Near saturation the remaining latency gap reflects queue growth. A fluid queue model driven only by a 4.4% throughput shortfall reproduces it, and shows that such a gap scales with the length of the benchmark run. What survives matching is a tail-latency advantage of 2.4 to 6.4x at equal throughput, consistent with cache-aware ordering of waiting requests. We derive six recommendations for comparing serving engines.
BibTeX
@misc{anwar2026controlled,
  author = {Muhammad Faizan Anwar and Sina Keshvadi},
  title = {{A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines}},
  year = {2026},
  eprint = {2611.05991},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05991},
}
Figure