TL;DR
This paper introduces a connected, end-to-end evaluation pipeline for deploying Large Language Model (LLM) inference platforms specifically for agentic workloads. By combining live benchmarking, high-fidelity trace replay, and synthetic workload generation, the authors demonstrate that replaying captured traces provides a far more stable and accurate basis for capacity sizing than live agent re-runs, which suffer from high non-deterministic variance.
What Problem Does This Paper Address?
As enterprises move away from expensive closed-model APIs toward self-hosted open-weight models (like Llama or DeepSeek) on stacks like vLLM, they face a unique challenge: agentic workloads. Unlike simple chat, agentic tasks involve multi-step reasoning, parallel tool calls, and contexts that grow turn-by-turn.
Current evaluation methods are fragmented. Quality frameworks measure if an agent succeeds but ignore the inference endpoint's performance. Conversely, load generators often use synthetic prompts or "hashed" traces that strip away the actual content, making it impossible to test features like prefix caching or Mixture-of-Experts (MoE) routing accurately. The authors identify a "validation gap" where sizing a deployment on unrealistic traces leads to capacity estimates that fail in production.
Methods
The authors propose an integrated five-stage lifecycle (Figure 1) that connects development to production:
- Small-scale testing: Functional API checks and initial tuning.
- Workload capture: Running real agents on benchmarks to record full-content traces (inputs, outputs, and tool definitions).
- Sizing: Replaying those captured traces at scale to find the saturation point.
- Pre-production: End-to-end acceptance testing under realistic contention.
- Production: Monitoring live traffic and feeding new traces back into the pipeline.
To enable this, they extended Exgentic for live benchmarking and Inference Perf for "dependency-aware" replay. Unlike standard replay, their system infers relationships between requests, allowing it to substitute live model outputs into downstream requests while keeping the agent's "trajectory" aligned with the original trace.
Datasets
The study utilizes several agentic benchmarks:
- HWE-Bench: Hardware bug repair tasks.
- CyberGym: Cybersecurity-related agent tasks.
- Terminal-Bench 2.1: Command-line interface interactions.
- SWE-Bench Multimodal: Software engineering tasks with image inputs.
- Public Traces: WEKA and NVIDIA Moontrace (used as baselines).
Crucially, the authors have open-sourced a new dataset containing 10,056 agent sessions with full request contents to facilitate further research.
Key Insights and Ideas
- The Drift Problem: Agents are inherently non-deterministic. If you try to "re-run" a benchmark to test performance, the agent might take a completely different path, leading to wild swings in token load. Replay solves this by enforcing the original tool choices and sequence.
- Workload Complementarity: Using code-coverage analysis on vLLM, the authors proved that you cannot rely on just one type of test. Live benchmarks, replayed traces, and synthetic workloads each hit unique Python code paths. For example, only a specific synthetic workload was able to trigger the "CPU-to-GPU restore" path in the KV cache manager.
- Public Traces are Too Optimistic: Publicly available traces often lack the specific "shape" of the target model's output. In their case study, sizing based on public traces overestimated throughput by 40% compared to traces captured directly from the target models.
Key Results
- Replay Fidelity: Replaying traces kept the 95th-percentile input token load within 1.18x of the original recording. In contrast, live re-runs drifted significantly, reaching up to 4.98x the original load.
- Consistency: Replay demonstrated high inter-run consistency, with a coefficient of variation for time-per-output-token (TPOT) between 5.2% and 10.7%, making it a reliable metric for performance regressions.
- Code Coverage: The combined pipeline reached 9,778 unique statements in the vLLM serving stack. The "live" family contributed the most unique coverage (331 statements), primarily by exercising specific API paths like Anthropic messages.
Limitations and Future Work
The authors acknowledge several limitations:
- Python-Centric: The coverage analysis only measures vLLM’s Python code, not the compiled C++/CUDA kernels where much of the heavy lifting happens.
- Single Configuration: The results are based on a specific vLLM version and hardware setup (H100/H200 GPUs); different configurations might yield different coverage patterns.
- Correctness vs. Coverage: A "covered" line of code means it was executed, but it doesn't guarantee the output was correct.
Future work includes scaling the generation of synthetic agentic workloads and further refining the automated regression analysis to better distinguish between true performance drops and normal variance.
Summary updated