TL;DR
The paper introduces Runway, a validation harness that transforms LLM inference benchmarking into a governed, auditable decision pipeline suitable for regulated industries like healthcare and finance. By wrapping NVIDIA’s AIPerf, Runway provides the documented evidence required by regulators while revealing that optimal engine and hardware choices are highly sensitive to concurrency and specific configurations.
What Problem Does This Paper Address?
In regulated industries (e.g., pharmaceutical manufacturing or medical devices), "faster is better" is not a sufficient justification for deploying a new AI model or inference engine. Regulators, such as the FDA under 21 CFR Part 11, require objective, documented evidence that a system meets specific contractual and legal obligations before it serves a single customer.
Standard benchmarking practices often treat engine selection as a one-time decision based on synthetic loads. However, LLM performance is notoriously fickle: it depends on "token shape" (input vs. output length), exhibits non-determinism, and behaves differently under varying levels of concurrency. The authors address the gap between raw performance measurement and the rigorous, traceable decision-making process required for "validated" production systems.
Methods
The authors developed Runway, a Python-based CLI tool organized around a "Before–Measure–After" architecture. The core philosophy is the separation of concerns: Runway handles the experiment design and the compliance logic, while delegating the actual measurement to an unmodified, industry-standard tool: NVIDIA AIPerf.
- Before: Runway defines "sweeps"—grids of variables including engines (vLLM, TensorRT-LLM), instance types, token shapes, and concurrency levels.
- Measure: Using Argo Workflows on Kubernetes (EKS), Runway deploys the inference engine as a sidecar container alongside AIPerf. This ensures the load is generated over
localhost, eliminating network latency from the measurement. - After: Runway evaluates the raw data against versioned SLA (Service Level Agreement) files. It applies a cost model based on live AWS pricing to determine not just which configuration is fastest, but which is most cost-effective.
A key technical innovation is Runway’s deterministic storage model. Every result is stored in a path that encodes exactly what was measured (dataset, model, engine, version). This allows the system to use caching as an audit trail: if a configuration has been tested before, the "cache hit" provides the historical evidence needed for a release decision without re-running the expensive GPU workload.
Datasets
The paper utilizes two primary types of workloads for evaluation:
- Synthetic Token Shapes: Generated by AIPerf to test specific combinations of input tokens (128 to 1024) and output tokens (128 to 2048).
- Internal Entity-Extraction Prompt Set: A real-world dataset used for "campaigns," featuring a shared system prefix of approximately 3,300 tokens and 200 output tokens.
- Models: The benchmarks were conducted using Qwen3-8B-FP8 and Qwen3.5-9B.
Key Insights and Ideas
- The Concurrency Crossover: Performance rankings are not static. An engine that dominates at low concurrency might fall behind as the load increases.
- The "Pricier is Cheaper" Paradox: Looking at hourly instance costs is misleading. A more expensive, high-performance GPU (like the NVIDIA Blackwell-based g7e) can actually reduce the cost per request by providing disproportionately higher throughput.
- Configuration > Engine Choice: The authors found that specific settings, such as speculative decoding or scheduler policies (e.g.,
MAX_UTILIZATIONvs.GUARANTEED_NO_EVICT), can shift performance more significantly than switching between major engines like vLLM and TensorRT-LLM. - Auditability via Caching: By making the storage path a function of the experiment parameters, the authors turned a standard engineering optimization (caching) into a regulatory feature (a retrievable, immutable audit trail).
Key Results
The authors reported three major findings from their Runway sweeps:
- Engine Inversion: In a comparison between vLLM and TensorRT-LLM on L40S GPUs, TensorRT-LLM was 35% faster at a concurrency of 1. However, at a concurrency of 100, vLLM took the lead with a 19% throughput advantage.
- Blackwell Efficiency: The newer
g7e.2xlarge(Blackwell) instance carries a 1.81x hourly price premium over theg6e.xlarge(L40S). However, because it roughly doubles throughput, the median cost per request was actually 1–15% lower on the "pricier" hardware. - Speculative Decoding Variance: Using Multi-Token Prediction (MTP) provided a 3x speed-up for TensorRT-LLM at low concurrency. However, for vLLM, MTP actually became a net performance cost at higher concurrency levels (above 16).
Limitations and Future Work
- Statistical Significance: Currently, Runway’s cache stores exactly one measurement per "cell." The authors acknowledge that this does not yet account for variance or provide confidence intervals, though a
--replicate=Nfeature is planned. - Synthetic vs. Real Traffic: While synthetic sweeps establish relative behavior, they do not yet perfectly mirror the specific distribution of production traffic at MasterControl.
- Availability: Runway is currently an internal tool at MasterControl and is not open-source, though the authors are open to inquiries from other practitioners in regulated spaces.
Summary updated