ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration

Anshu Raina, Yao Fu, Zhang Jiejing, Peyman Razaghi, Shiksha Patel, Ravi Gupta, Deval Shah, Adeem Jassani, Subhajit Dutta Chowdhury, Lorri Rao, Kailash Gogineni, Zhenyu Gu, Onil Gunawardana, Emad BarsoumarXiv 2026 · arXiv

Finding the "Goldilocks" configuration for serving a Large Language Model (LLM) is a nightmare for infrastructure engineers. Should you use Tensor Parallelism (TP) or Expert Parallelism (EP)? Is vLLM faster than SGLang for your specific workload? Does disaggregating prefill and decode actually pay off?

Answering these questions usually requires exhaustive trial and error. In a recent paper, researchers from AMD introduce ROCm InferaSim, a framework that slashes these trial costs by orders of magnitude, turning a 25,000 GPU-hour search problem into a sub-10-hour measurement task.

TL;DR

ROCm InferaSim predicts LLM serving performance by benchmarking a few "anchor" configurations and using analytical scaling laws to project results across a massive search space. It achieves a median error of just 8.1% on latency metrics while reducing the required GPU-hours for evaluation by over 2,500x.

What Problem Does This Paper Address?

The state-space for LLM deployment is exploding. Operators must choose between different serving engines, parallelism strategies (TP, EP, PP, DP), numeric precisions, and scheduler settings. Because these choices interact in complex ways with specific hardware (like the AMD Instinct™ MI355X) and workloads (like multi-turn coding agents), the "best" configuration is a moving target.

Currently, the industry standard is "guess and check." However, a single benchmark run can take minutes. Exhaustively searching the tens of thousands of possible configurations for a modern model like DeepSeek-V4 would cost a fortune in compute time. Existing simulators often fall short because they either require profiling every single operator (which breaks when kernels change) or use overly simplistic "roofline" models that ignore the fixed overheads of kernel launches and synchronization.

Methods

The core philosophy of ROCm InferaSim is: Measure kernels once, scale analytically.

1. Anchors and Transfer Laws

The researchers observed that many configuration changes (like batch size or GPU count) change how much work a GPU does, but not which kernels it runs. They define a "kernel set" as the collection of kernels launched in one engine step.

  • Anchors: They benchmark each unique kernel set once on a single node.
  • Transfer Laws: They use analytical formulas to scale these "anchors." Crucially, they keep the fixed per-step costs (launch overhead, scheduling) constant while scaling the divisible GPU work. This fixes the 44% error seen in naive scaling models.

2. Reduce-Measure-Restore

For massive deployments spanning multiple nodes, the framework uses a three-step dance:

  1. Reduce the target to a single-node equivalent that runs the same kernels.
  2. Measure that run to create an anchor.
  3. Restore the multi-node overheads (network hops, pipeline bubbles) using measured constants.
The ROCm InferaSim architecture showing the flow from serving specs to the final search.

3. Discrete-Event Simulation (DES)

To capture tail latency (p99s), which averages can't predict, they feed their step-cost models into a DES. This simulator mimics engine-specific schedulers, prefix caching, and the "think time" of agentic users.

4. The Tuning Agent

Finally, they wrap this in an LLM-guided tuning agent (using DSPy) that searches the predicted space to find the Pareto frontier of throughput vs. latency.

Datasets

The framework was evaluated using:

  • Fixed-length workloads: 172 configurations across eight models (e.g., Llama-3.1-8B, Qwen3, DeepSeek-V4) with varying prompt/output lengths.
  • AgentX: A multi-turn agentic coding trace from SemiAnalysis, used to test realistic traffic patterns with prefix reuse.
  • InferenceX: A public collection of measured serving runs used to validate disaggregated prefill/decode predictions.

Key Insights and Ideas

  • Fixed Costs Matter: Naive models assume doubling GPUs halves latency. ROCm InferaSim proves that fixed kernel launch and synchronization overheads stay constant, becoming a larger bottleneck as you shard the model further.
  • The "Anchor" Abstraction: By anchoring on the kernel set rather than the operator, the tool remains robust even when engines or libraries are updated, as long as the underlying kernels are similar.
  • Simulating the Scheduler is Non-Negotiable: You cannot predict p99 tail latency without a discrete-event simulator that understands how new prompts stall existing decodes.
  • Disaggregation is a Memory Problem: In long-context agentic traces, the decision to split prefill and decode is often driven by KV-cache memory pressure rather than just compute throughput.

Key Results

The efficiency gains are staggering. As shown in Figure 1, the "GPU-hours to measure" drops from a projected 25,000 hours for an exhaustive search to effectively zero (using CPU-based simulation) once the initial 9.1 hours of anchor measurements are done.

Comparison of GPU-hours required for measurement and the resulting median error across different validation sets.
  • Accuracy: Median error for Time Per Output Token (TPOT) was 8.1%. Throughput for complex agentic workloads was predicted within 8%.
  • Zero-Shot Disaggregation: The model predicted the performance of disaggregated deployments (splitting prefill and decode GPUs) within 5.5% error, even though it never used a disaggregated run as an anchor.
  • Search Fidelity: The tuning agent's "top 5" list consistently contained configurations within 12% of the absolute best possible hardware-measured performance.

Limitations and Future Work

  • Hardware Scope: All measurements were performed on AMD Instinct MI355X GPUs. While the transfer laws are general, the constants (like network latency and kernel launch times) must be re-measured for other architectures like NVIDIA's H100.
  • Kernel Changes: If a configuration change alters the kernel set (e.g., switching from FP8 to MXFP4 or changing the attention implementation), a new anchor must be measured.
  • Future Directions: The authors suggest extending the evaluation to NVIDIA GPUs and exploring more complex "prefill-as-a-service" architectures that span multiple datacenters.

Summary updated

Abstract

Serving an LLM efficiently depends on the engine, how the model is split across GPUs, precision, scheduler settings, speculative decoding, KV-cache management, and prefill-decode disaggregation. The best configuration depends on the model, hardware, workload, and objective. Operators usually find it by trial and error. Exhaustively measuring the candidates in our study would take about 25,000 GPU-hours. ROCm InferaSim predicts serving performance from a few short measurements. Choices such as batch size, prompt length, and GPU count change the amount of work per GPU without changing the kernels. ROCm InferaSim benchmarks each distinct set of kernels once, on at most one node. We call this measurement an anchor. Analytical laws carry that anchor to other deployments that run the same kernels. For multi-node deployments, ROCm InferaSim reduces the configuration to a single-node equivalent, measures it, and scales the result analytically. A simulator replays multi-turn traffic, and a tuning agent searches configurations using the predictions. On AMD Instinct MI355X GPUs with vLLM, SGLang, and ATOM, 11 anchors took 9.1 GPU-hours, against 204 GPU-hours to measure the 226 fixed-length configurations they cover. On 172 configurations, time per output token has 8.1% median error. On a multi-turn coding workload, throughput is within 8%. For 159 disaggregated deployments, none used as an anchor, throughput is within 5.5%. A five-candidate short list contains a deployment within 12% of the best feasible pairing of model and latency target.
BibTeX
@misc{raina2026rocm,
  author = {Anshu Raina and Yao Fu and Zhang Jiejing and Peyman Razaghi and Shiksha Patel and Ravi Gupta and Deval Shah and Adeem Jassani and Subhajit Dutta Chowdhury and Lorri Rao and Kailash Gogineni and Zhenyu Gu and Onil Gunawardana and Emad Barsoum},
  title = {{ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration}},
  year = {2026},
  eprint = {2611.04032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04032},
}
Figure