Can Agentic LLMs Write Task-Based Parallel Code? A Study of OpenMP Tasking, HPX, StarPU and TTG

Linus Bantel, Anna-Lena Roth, Tim Thüring, Jonas Posner, Dirk PflügerarXiv 2026 · arXiv

TL;DR

Researchers evaluated whether LLM agents—models that can edit, compile, and test their own code—can master complex C++ task-based parallel programming. While capable models achieved a 57–82% success rate and sometimes outperformed industry-standard libraries like OpenBLAS, they frequently fell into "correctness traps" where code passed functional tests but lacked actual parallelism or suffered from poor scaling.

What Problem Does This Paper Address?

Writing high-performance parallel code is notoriously difficult. Unlike simple loop-level parallelism (like basic OpenMP), task-based programming allows developers to define a graph of dependent tasks that a runtime system executes asynchronously. This is more powerful but requires managing complex dependencies and learning niche APIs like HPX, StarPU, or Template Task Graph (TTG).

Previous studies showed that LLMs struggle with these frameworks in "one-shot" prompts, often hallucinating outdated APIs. This paper investigates if an agentic workflow—where the LLM acts as a developer by iterating through a cycle of writing, compiling, and debugging—can overcome these hurdles to produce correct and performant HPC (High-Performance Computing) code.

Methods

The authors used an open-source coding agent called opencode paired with five different open-weight LLMs (including variants of Qwen and Nemotron).

The workflow was "zero-shot" in terms of custom tools: the agent was given a standard environment (GCC, CMake, and the parallel runtimes) and a task prompt. The agent's goal was to implement a specific kernel that matched the output of a provided serial reference. Crucially, the agent could:

  1. Read/Write files.
  2. Execute shell commands to compile.
  3. Run the code against sample inputs to verify correctness.
  4. Iterate for up to two hours to fix errors.

The generated code was then independently evaluated for correctness across multiple thread counts, strong scaling efficiency, and peak memory usage.

Datasets

The study utilized seven classic HPC benchmarks (kernels) of varying complexity:

  • Basic: Pi approximation (reduction), AXPY, and GEMV (memory-bound).
  • Compute-Intensive: Matrix-matrix multiplication (GEMM).
  • Complex Dependencies: Cholesky and LU factorization, and a Linear Solve (tiled dense linear algebra).

These were implemented across four frameworks: OpenMP Tasking, HPX, StarPU, and TTG, plus a serial baseline.

Key Insights and Ideas

  • The Agentic Loop is a Game Changer: For capable models, the ability to see compiler errors and test output resolved most API hallucinations. 90% of kernels that the agent managed to build at least once eventually passed the correctness check.
  • Model Capability is the Bottleneck: There is a massive "intelligence threshold." The strongest models (Qwen-3.8-27B and Ornith-35B) were highly successful, while smaller or less capable models (like Nemotron-3-Nano-4B) failed to even compile their code, often ending sessions without writing a single line.
  • The "Correctness Trap": This is the paper's most vital warning. An agent's only metric for success is "does the output match the reference?" Consequently, several agents produced "parallel" code that was actually serial (e.g., wrapping the entire algorithm in a single task). Because it produced the right answer, the agent stopped iterating, unaware that the performance was abysmal.
  • Data Scarcity Matters: Success rates tracked closely with the popularity of the framework. OpenMP and HPX (well-documented) were handled reliably, while TTG (the youngest and least documented) saw the most timeouts and crashes.

Key Results

  • Success Rates: The three most capable models produced correct, genuinely task-parallel kernels in 57–82% of cases.
  • Performance: In several instances (GEMM and factorizations), the LLM-generated code matched or exceeded the performance of the multithreaded OpenBLAS/LAPACK reference on a 32-core system.
  • Scaling: While the Pi approximation scaled nearly ideally (speedups of ~60x on 64 cores), more complex kernels often suffered from "serialized task graphs" or "unsuitable task granularity," where the tasks were either too small (adding overhead) or forced to run one after another.
  • Root Causes of Failure: For the 42 kernels that failed, the primary culprits were non-existent APIs (hallucinations) and algorithm/indexing errors (like integer truncation).

Limitations and Future Work

The authors identify a major limitation in the current agentic setup: agents do not measure performance. Because the agents were only instructed to verify correctness, they often settled for inefficient code.

Future Work should focus on:

  1. Performance-Aware Agents: Giving agents tools to profile code and measure scaling, allowing them to iterate until the code is both correct and fast.
  2. Niche Framework Support: Improving LLM performance on frameworks like TTG where training data is scarce, perhaps through better RAG (Retrieval-Augmented Generation) with up-to-date documentation.
  3. Granularity Control: Teaching models to better choose task sizes to balance parallel overhead with CPU utilization.

Summary updated

Abstract

Task-based programming offers more expressive power than loop-level parallelism, but the runtimes providing it differ widely in API style, maturity, and the amount of public code to learn from. This paper asks how well LLM agents write C++ for four of them: OpenMP Tasking, HPX, StarPU, and TTG. Instead of single one-shot prompts, we use an agentic workflow in which the model edits files, compiles, and runs its kernels until it reproduces a reference output. Five locally hosted open-weight LLM implement seven HPC kernels, including matrix multiplication, Cholesky and LU factorization, in all four frameworks and a serial baseline to verify their understanding of the problem in general. Every kernel is evaluated independently for correctness at each concurrency level, strong scaling, and peak memory. Model capability dominates the outcome: three models produce correct, genuinely task-parallel kernels in 57-82% of the cases, while two models rarely compile their own code and almost never succeed. For the capable models, OpenMP Tasking and HPX are handled most reliably, StarPU is close behind, and TTG trails. The best generated GEMM and factorization kernels match or exceed the multithreaded OpenBLAS/LAPACK reference. However, correctness overstates parallel quality: several "parallel" kernels contain no parallelism at all; and serialized task graphs and unsuitable task granularity remain common.
BibTeX
@misc{bantel2026can,
  author = {Linus Bantel and Anna-Lena Roth and Tim Thüring and Jonas Posner and Dirk Pflüger},
  title = {{Can Agentic LLMs Write Task-Based Parallel Code? A Study of OpenMP Tasking, HPX, StarPU and TTG}},
  year = {2026},
  eprint = {2611.19105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19105},
}
Figure