Training a state-of-the-art Large Language Model (LLM) is an exercise in extreme resource management. With training costs for models like DeepSeek-V3 reaching millions of dollars, engineers can't afford to "guess and check" their parallelization strategies. Performance simulators offer a cheaper way to find the optimal configuration, but until now, the field has lacked a standardized way to compare these simulators.
Enter LM-SimBench, a new unified framework, dataset, and benchmark designed to bring order to the world of LLM training simulation.
TL;DR
LM-SimBench introduces a standardized framework and a massive profiling dataset (348 configurations across 10 LLM families) to evaluate LLM training simulators. It enables researchers to rigorously compare different simulation approaches and demonstrates that performance data from one hardware platform (like Ascend NPUs) can effectively guide configuration choices on another (like NVIDIA GPUs).
What Problem Does This Paper Address?
Finding the "sweet spot" for distributed training involves balancing five dimensions of parallelism: data (DP), tensor (TP), pipeline (PP), context (CP), and expert (EP). Poor alignment leads to Out-of-Memory (OOM) errors, hardware bottlenecks, or low utilization—all of which waste precious compute time.
While many simulators exist, they are notoriously difficult to compare. One might validate against A100 GPUs using Megatron-DeepSpeed, while another uses V100s and NCCL measurements. This "apples-to-oranges" landscape makes it impossible to know which simulator is actually the most accurate or useful for a given workload.
Methods
The authors propose a unified simulation pipeline that decomposes the process into five functional stages:
- Model Compilation: Partitioning the abstract model based on the parallel strategy.
- Device Binding: Assigning model components to specific hardware devices and communication groups.
- Computation Process Modeling: Constructing an execution representation (like a task graph or analytical phases).
- Operator-Level Estimation: Predicting the cost of individual computation and communication units.
- Schedule-Level Aggregation: Combining unit costs to predict the final iteration time and memory usage.
This framework allows researchers to swap out different cost estimators (analytical vs. profiling-based) while keeping the rest of the pipeline constant, enabling fair comparisons.
Datasets
The paper introduces the LM-SimBench Dataset, a comprehensive collection of training profiles:
- Scope: 23 model settings from 10 dense and Mixture-of-Experts (MoE) families (e.g., Qwen, Llama, InternLM).
- Scale: 348 feasible parallel configurations.
- Hardware: Primary data collected on Ascend 910B3 NPUs; a cross-platform supplement includes matched NVIDIA V100 GPU measurements.
- Granularity: Six linked views ranging from high-level iteration time down to individual matrix-kernel latencies.
Key Insights and Ideas
- The Speed-Accuracy Trade-off: The benchmark compares analytical simulators (like Calculon) against profiling-based ones (like vTrain). Analytical models are lightning-fast (14.3x speedup) but less accurate, while profiling-based models offer superior precision at the cost of needing pre-collected data.
- Cross-Platform Transferability: One of the most surprising findings is that NPU-based profiles are highly effective at guiding GPU configuration. Even though absolute runtimes differ, the relative ranking of configurations remains consistent across hardware stacks.
- Diagnosis via Multi-Granularity: By linking end-to-end time to operator-level traces, the benchmark helps identify why a simulator fails. For instance, it revealed that some simulators struggle because they assume serialized communication, missing the performance gains from computation-communication overlap.
Key Results
- Simulation Accuracy: In head-to-head tests, the profiling-based
vTrainachieved a remarkably low 1.86% error (sMAPE) in iteration time prediction, while the analyticalCalculonhad an 8.88% error. - Configuration Selection: Both simulators successfully identified the fastest configuration in the test set, showing that even less-accurate simulators can be highly effective for "ranking" strategies.
- NPU-to-GPU Guidance: Using NPU data to create a "Top-3" shortlist for GPU deployment consistently captured the optimal GPU configuration. Selecting the NPU-fastest configuration directly resulted in a GPU runtime penalty of 0.0% to 6.9%, proving the utility of cross-platform insights.
Limitations and Future Work
The authors acknowledge that their current dataset focuses on decoder-only Transformers and training workloads. They suggest that future work should extend this to broader model architectures and include inference/serving scenarios. Additionally, while the NPU-to-GPU transfer is promising, expanding the benchmark to include more diverse accelerators (like H100s or specialized AI chips) would further validate the framework's universality.
Summary updated