Finding the "Goldilocks" configuration for serving a Large Language Model (LLM) is a nightmare for infrastructure engineers. Should you use Tensor Parallelism (TP) or Expert Parallelism (EP)? Is vLLM faster than SGLang for your specific workload? Does disaggregating prefill and decode actually pay off?
Answering these questions usually requires exhaustive trial and error. In a recent paper, researchers from AMD introduce ROCm InferaSim, a framework that slashes these trial costs by orders of magnitude, turning a 25,000 GPU-hour search problem into a sub-10-hour measurement task.
TL;DR
ROCm InferaSim predicts LLM serving performance by benchmarking a few "anchor" configurations and using analytical scaling laws to project results across a massive search space. It achieves a median error of just 8.1% on latency metrics while reducing the required GPU-hours for evaluation by over 2,500x.
What Problem Does This Paper Address?
The state-space for LLM deployment is exploding. Operators must choose between different serving engines, parallelism strategies (TP, EP, PP, DP), numeric precisions, and scheduler settings. Because these choices interact in complex ways with specific hardware (like the AMD Instinct™ MI355X) and workloads (like multi-turn coding agents), the "best" configuration is a moving target.
Currently, the industry standard is "guess and check." However, a single benchmark run can take minutes. Exhaustively searching the tens of thousands of possible configurations for a modern model like DeepSeek-V4 would cost a fortune in compute time. Existing simulators often fall short because they either require profiling every single operator (which breaks when kernels change) or use overly simplistic "roofline" models that ignore the fixed overheads of kernel launches and synchronization.
Methods
The core philosophy of ROCm InferaSim is: Measure kernels once, scale analytically.
1. Anchors and Transfer Laws
The researchers observed that many configuration changes (like batch size or GPU count) change how much work a GPU does, but not which kernels it runs. They define a "kernel set" as the collection of kernels launched in one engine step.
- Anchors: They benchmark each unique kernel set once on a single node.
- Transfer Laws: They use analytical formulas to scale these "anchors." Crucially, they keep the fixed per-step costs (launch overhead, scheduling) constant while scaling the divisible GPU work. This fixes the 44% error seen in naive scaling models.
2. Reduce-Measure-Restore
For massive deployments spanning multiple nodes, the framework uses a three-step dance:
- Reduce the target to a single-node equivalent that runs the same kernels.
- Measure that run to create an anchor.
- Restore the multi-node overheads (network hops, pipeline bubbles) using measured constants.
3. Discrete-Event Simulation (DES)
To capture tail latency (p99s), which averages can't predict, they feed their step-cost models into a DES. This simulator mimics engine-specific schedulers, prefix caching, and the "think time" of agentic users.
4. The Tuning Agent
Finally, they wrap this in an LLM-guided tuning agent (using DSPy) that searches the predicted space to find the Pareto frontier of throughput vs. latency.
Datasets
The framework was evaluated using:
- Fixed-length workloads: 172 configurations across eight models (e.g., Llama-3.1-8B, Qwen3, DeepSeek-V4) with varying prompt/output lengths.
- AgentX: A multi-turn agentic coding trace from SemiAnalysis, used to test realistic traffic patterns with prefix reuse.
- InferenceX: A public collection of measured serving runs used to validate disaggregated prefill/decode predictions.
Key Insights and Ideas
- Fixed Costs Matter: Naive models assume doubling GPUs halves latency. ROCm InferaSim proves that fixed kernel launch and synchronization overheads stay constant, becoming a larger bottleneck as you shard the model further.
- The "Anchor" Abstraction: By anchoring on the kernel set rather than the operator, the tool remains robust even when engines or libraries are updated, as long as the underlying kernels are similar.
- Simulating the Scheduler is Non-Negotiable: You cannot predict p99 tail latency without a discrete-event simulator that understands how new prompts stall existing decodes.
- Disaggregation is a Memory Problem: In long-context agentic traces, the decision to split prefill and decode is often driven by KV-cache memory pressure rather than just compute throughput.
Key Results
The efficiency gains are staggering. As shown in Figure 1, the "GPU-hours to measure" drops from a projected 25,000 hours for an exhaustive search to effectively zero (using CPU-based simulation) once the initial 9.1 hours of anchor measurements are done.
- Accuracy: Median error for Time Per Output Token (TPOT) was 8.1%. Throughput for complex agentic workloads was predicted within 8%.
- Zero-Shot Disaggregation: The model predicted the performance of disaggregated deployments (splitting prefill and decode GPUs) within 5.5% error, even though it never used a disaggregated run as an anchor.
- Search Fidelity: The tuning agent's "top 5" list consistently contained configurations within 12% of the absolute best possible hardware-measured performance.
Limitations and Future Work
- Hardware Scope: All measurements were performed on AMD Instinct MI355X GPUs. While the transfer laws are general, the constants (like network latency and kernel launch times) must be re-measured for other architectures like NVIDIA's H100.
- Kernel Changes: If a configuration change alters the kernel set (e.g., switching from FP8 to MXFP4 or changing the attention implementation), a new anchor must be measured.
- Future Directions: The authors suggest extending the evaluation to NVIDIA GPUs and exploring more complex "prefill-as-a-service" architectures that span multiple datacenters.
Summary updated