LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

Yu Dong, Hong Li, Qiang Sun, Xu Zhou, Jinbiao Huo, Qixiu Cheng, Zhiyuan LiuarXiv 2026 · arXiv

Training a state-of-the-art Large Language Model (LLM) is an exercise in extreme resource management. With training costs for models like DeepSeek-V3 reaching millions of dollars, engineers can't afford to "guess and check" their parallelization strategies. Performance simulators offer a cheaper way to find the optimal configuration, but until now, the field has lacked a standardized way to compare these simulators.

Enter LM-SimBench, a new unified framework, dataset, and benchmark designed to bring order to the world of LLM training simulation.

TL;DR

LM-SimBench introduces a standardized framework and a massive profiling dataset (348 configurations across 10 LLM families) to evaluate LLM training simulators. It enables researchers to rigorously compare different simulation approaches and demonstrates that performance data from one hardware platform (like Ascend NPUs) can effectively guide configuration choices on another (like NVIDIA GPUs).

What Problem Does This Paper Address?

Finding the "sweet spot" for distributed training involves balancing five dimensions of parallelism: data (DP), tensor (TP), pipeline (PP), context (CP), and expert (EP). Poor alignment leads to Out-of-Memory (OOM) errors, hardware bottlenecks, or low utilization—all of which waste precious compute time.

Figure 1. Simulation for model–resource alignment. Effective alignment improves utilization and reduces cost; poor alignment creates bottlenecks and resource waste. Simulation estimates these effects before deployment, reducing the need for repeated full-scale trials.

While many simulators exist, they are notoriously difficult to compare. One might validate against A100 GPUs using Megatron-DeepSpeed, while another uses V100s and NCCL measurements. This "apples-to-oranges" landscape makes it impossible to know which simulator is actually the most accurate or useful for a given workload.

Methods

The authors propose a unified simulation pipeline that decomposes the process into five functional stages:

  1. Model Compilation: Partitioning the abstract model based on the parallel strategy.
  2. Device Binding: Assigning model components to specific hardware devices and communication groups.
  3. Computation Process Modeling: Constructing an execution representation (like a task graph or analytical phases).
  4. Operator-Level Estimation: Predicting the cost of individual computation and communication units.
  5. Schedule-Level Aggregation: Combining unit costs to predict the final iteration time and memory usage.
Figure 2. Unified pipeline for large-model training simulation. Model, training, and system configurations drive compilation, device binding, and computation-process modeling. Computation and communication costs from operator-level estimation feed schedule-level aggregation to predict iteration time. Profiling supplies measured costs; the optimizer uses predictions for configuration search.

This framework allows researchers to swap out different cost estimators (analytical vs. profiling-based) while keeping the rest of the pipeline constant, enabling fair comparisons.

Datasets

The paper introduces the LM-SimBench Dataset, a comprehensive collection of training profiles:

  • Scope: 23 model settings from 10 dense and Mixture-of-Experts (MoE) families (e.g., Qwen, Llama, InternLM).
  • Scale: 348 feasible parallel configurations.
  • Hardware: Primary data collected on Ascend 910B3 NPUs; a cross-platform supplement includes matched NVIDIA V100 GPU measurements.
  • Granularity: Six linked views ranging from high-level iteration time down to individual matrix-kernel latencies.

Key Insights and Ideas

  • The Speed-Accuracy Trade-off: The benchmark compares analytical simulators (like Calculon) against profiling-based ones (like vTrain). Analytical models are lightning-fast (14.3x speedup) but less accurate, while profiling-based models offer superior precision at the cost of needing pre-collected data.
  • Cross-Platform Transferability: One of the most surprising findings is that NPU-based profiles are highly effective at guiding GPU configuration. Even though absolute runtimes differ, the relative ranking of configurations remains consistent across hardware stacks.
  • Diagnosis via Multi-Granularity: By linking end-to-end time to operator-level traces, the benchmark helps identify why a simulator fails. For instance, it revealed that some simulators struggle because they assume serialized communication, missing the performance gains from computation-communication overlap.

Key Results

  • Simulation Accuracy: In head-to-head tests, the profiling-based vTrain achieved a remarkably low 1.86% error (sMAPE) in iteration time prediction, while the analytical Calculon had an 8.88% error.
  • Configuration Selection: Both simulators successfully identified the fastest configuration in the test set, showing that even less-accurate simulators can be highly effective for "ranking" strategies.
  • NPU-to-GPU Guidance: Using NPU data to create a "Top-3" shortlist for GPU deployment consistently captured the optimal GPU configuration. Selecting the NPU-fastest configuration directly resulted in a GPU runtime penalty of 0.0% to 6.9%, proving the utility of cross-platform insights.
Figure 3. NPU profiles guide GPU configuration selection. Each point is a paired configuration ranked by measured iteration time (rank 1 is fastest). The shaded NPU Top-3 shortlist contains the GPU optimum (star) for both models. The dashed line denotes identical ranks.

Limitations and Future Work

The authors acknowledge that their current dataset focuses on decoder-only Transformers and training workloads. They suggest that future work should extend this to broader model architectures and include inference/serving scenarios. Additionally, while the NPU-to-GPU transfer is promising, expanding the benchmark to include more diverse accelerators (like H100s or specialized AI chips) would further validate the framework's universality.

Summary updated

Abstract

Evaluating candidate parallelization strategies through distributed training runs makes strategy selection prohibitively expensive at scale. Performance simulation offers a lower cost alternative, yet controlled comparisons among existing simulators remain difficult because they differ in the training stages they model and the workloads on which they are evaluated. We introduce LM-SimBench, a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. The framework decomposes simulation into functional stages from model compilation to schedule level aggregation, connecting execution representations and cost estimators to measurable evaluation targets. Following this framework, we profile 23 model settings from 10 dense and Mixture-of-Experts (MoE) LLM families on an Ascend Neural Processing Unit (NPU) cluster. The resulting dataset contains 348 feasible configurations that jointly vary five parallelism dimensions and activation recomputation, with six views spanning configuration, rank, operator, event, and kernel granularity. Evaluations of representative training simulators demonstrate how LM-SimBench supports comparisons of prediction accuracy, assessment of configuration selection, and diagnosis of prediction errors. Matched GPU-NPU experiments validate the cross-platform applicability of the collection and evaluation protocol and demonstrate NPU-guided GPU configuration selection.
BibTeX
@misc{dong2026lm,
  author = {Yu Dong and Hong Li and Qiang Sun and Xu Zhou and Jinbiao Huo and Qixiu Cheng and Zhiyuan Liu},
  title = {{LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation}},
  year = {2026},
  eprint = {2611.14285},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14285},
}
Figure