SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation

Yibo He, Shuoran Zhao, Jiaming Huang, Yingjie Fu, Hao Yu, Cunjian Huang, Tao XiearXiv 2025 · DOI · arXiv

Introduction

To achieve peak performance in modern computing, developers increasingly rely on Single Instruction Multiple Data (SIMD) architectures. SIMD allows processors to execute a single operation across multiple data points simultaneously, providing a critical performance boost for data-intensive tasks. While modern compilers attempt to "auto-vectorize" code, they frequently fail due to a lack of compile-time information, forcing developers to manually write code using SIMD intrinsics—low-level C/C++ functions that map directly to specific CPU instructions.

Writing intrinsic code is a specialized and error-prone task. It requires deep knowledge of hardware-specific function names (e.g., _mm256_add_pd), strict memory alignment, and complex control flow management. While Large Language Models (LLMs) have transformed general software engineering, their capability to handle "bare metal" requirements has remained unquantified. This paper introduces SimdBench, the first comprehensive benchmark designed to evaluate LLM proficiency in generating vectorized code across x86, ARM, and RISC-V architectures.

The SimdBench Framework

SimdBench consists of 136 high-quality programming tasks designed to test the limits of LLM code generation. The benchmark covers five major SIMD extensions:

  • x86: SSE and AVX
  • ARM: Neon and SVE (Scalable Vector Extension)
  • RISC-V: RVV

The tasks were constructed using a two-pronged approach. First, 62 tasks were hand-crafted to focus on fundamental operations such as data movement, mathematical logic, and bitwise manipulation. Second, 74 tasks were adapted from the HumanEval-CPP benchmark, specifically filtered for problems that inherently benefit from parallelization.

To ensure the validity of the generated code, SimdBench employs a rigorous evaluation pipeline. This includes differential testing, which compares the LLM's SIMD output against a trusted scalar solution for correctness, and performance measurement using the Google Benchmark library on physical hardware (AWS Graviton for ARM and AMD EPYC for x86).

The "Vectorization Gap"

The study reveals a "universal decrease" in performance when LLMs transition from scalar to SIMD code. This "Vectorization Gap" is stark: for instance, the DeepSeek-R1 model achieves a ~92% pass rate on standard scalar code but drops to just ~19% when tasked with RISC-V vector code.

This performance degradation is closely tied to architecture bias. LLMs perform significantly better on x86 (SSE/AVX) and ARM Neon than on newer standards like SVE or RVV. This disparity likely stems from the training data; x86 has decades of documented code available, whereas newer "vector-length agnostic" standards are less represented in public repositories.

Error Analysis and Hallucinations

The evaluation identified two primary failure modes in LLM-generated SIMD code:

  1. Instruction-Level Hallucinations (40.68%): The most frequent error was the use of "undeclared identifiers." LLMs often guess intrinsic function names or use outdated prefixes (such as omitting the required __riscv_ prefix for RISC-V), suggesting that models have not fully internalized the latest hardware documentation.
  2. Logical and Alignment Bugs (26.26%): Even when the syntax was correct, models struggled with the logic of vectorization. Common issues included failing to handle data alignment requirements or incorrectly managing "loop remainders"—the leftover data points that do not fit into a full vector register.

Key Results and Top Performers

Despite these challenges, the study found that when LLMs successfully generate SIMD code, the performance gains are substantial. Valid SIMD samples frequently outperformed scalar code, though the advantage narrows when compared against compilers set to high optimization levels (e.g., -O3).

Among the 18 models tested—including proprietary models like GPT-4o and Claude 3.5 Sonnet—DeepSeek-R1 emerged as the top performer. As a reasoning-focused model, DeepSeek-R1 achieved an average pass@5 of 75.44% across all five intrinsic types, showing particular resilience on the difficult SVE and RVV tasks where other models failed entirely.

Future Directions

The findings from SimdBench suggest several paths for improving LLM utility in systems engineering:

  • Retrieval-Augmented Generation (RAG): Providing models with real-time access to official Intel, ARM, or RISC-V intrinsic manuals could mitigate the high rate of name hallucinations.
  • Hardware-Aware Chain-of-Thought: Encouraging models to first draft a scalar solution and then systematically refactor it into SIMD code may reduce logical errors.
  • Dataset Curation: There is a pressing need to update training sets with modern, well-documented SIMD examples to overcome the "data staleness" affecting newer architectures.

SimdBench is now open-sourced, providing a necessary benchmark for the next generation of LLMs aimed at high-performance computing and low-level systems programming.

Summary updated

BibTeX
@misc{he2025simdbench,
  author = {Yibo He and Shuoran Zhao and Jiaming Huang and Yingjie Fu and Hao Yu and Cunjian Huang and Tao Xie},
  title = {{SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation}},
  year = {2025},
  eprint = {2507.15224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.15224},
}

Loading related content...

Figure