CONCUR: Benchmarking LLMs for Concurrent Code Generation

Jue Huang, Tarek Mahmud, Corina S. Pasareanu, Guowei YangSubmission to ICLR 2026 · Rejected

TL;DR

While Large Language Models (LLMs) have become surprisingly good at writing sequential code, they have a massive blind spot: concurrency. This paper introduces CONCUR, the first benchmark specifically designed to evaluate LLMs on multi-threaded programming using formal model checking to catch non-deterministic bugs like deadlocks and race conditions that standard tests miss.

What Problem Does This Paper Address?

Writing code that does one thing at a time is relatively straightforward. Writing code where multiple threads interact simultaneously is a nightmare. Concurrent programming introduces "non-determinism"—the idea that a program might work perfectly 99 times but crash on the 100th because of a specific, rare interleaving of tasks.

Existing benchmarks like HumanEval or MBPP focus almost exclusively on sequential logic. They rely on unit tests or static similarity metrics (like CodeBLEU) to judge correctness. However, a concurrent program can pass a unit test simply because the "bad" thread schedule didn't happen during that specific run. Furthermore, static metrics only check if the code looks right, not if it behaves right under pressure. The authors argue that evaluating LLMs only on sequential code leaves a critical gap in our understanding of their true software engineering capabilities.

Methods

The researchers moved beyond simple "run and see" testing by employing Formal Methods. Specifically, they used a tool called Java PathFinder (JPF), an explicit-state model checker.

Unlike a standard JVM that just executes code once, JPF systematically explores all possible thread interleavings and execution paths within a defined bound. This allows the benchmark to rigorously detect:

  • Deadlocks: Threads waiting on each other forever.
  • Race Conditions: Multiple threads accessing shared data without proper synchronization.
  • Starvation: High-priority threads hogging resources.
  • Uncaught Exceptions: Errors that only trigger under specific concurrent conditions.

The evaluation framework is a two-stage process: first, it checks if the code compiles in a Java 8 environment; then, it hands the successful builds to JPF for deep behavioral verification.

Datasets

The authors introduced the CONCUR benchmark, which consists of 115 concurrency problems:

  • 43 Base Problems: Curated from the classic textbook Java Concurrency in Practice (Goetz, 2006). These cover fundamental constructs like synchronized blocks, ReentrantLock, Semaphore, and ExecutorService.
  • 72 Mutant Variants: To prevent models from simply "memorizing" textbook solutions, the authors used Gemini to generate variations of the base problems, which were then manually validated for semantic correctness.

All problems are paired with structured prompts and validated ground-truth implementations in Java.

Key Insights and Ideas

  • The "Conceptual vs. Implementation" Gap: One of the most interesting findings was the "Single Thread" error. LLMs often wrote perfectly thread-safe classes (like a thread-safe counter) but then wrote a main method that only used a single thread. They "understood" the concept of thread safety but failed to implement the actual concurrency required by the prompt.
  • CodeBLEU is Unreliable for Concurrency: The study found that CodeBLEU scores (which measure how much the generated code "looks like" the reference code) correlate very poorly with actual correctness. A model could have a high CodeBLEU score while containing a fatal deadlock that JPF easily caught.
  • The Power of Multiple Attempts: Mid-performing models saw a massive boost when moving from pass@1 (one attempt) to pass@3 (three attempts). For example, codeqwen:7b more than doubled its accuracy. This suggests that while these models "know" the right patterns, they are inconsistent in applying them.
  • Syntax is Still a Hurdle: Even in 2026, syntax errors remain the leading cause of failure (71% of compilation errors), often because LLMs try to use third-party libraries despite being told to use only standard Java 8.

Key Results

The researchers evaluated 23 state-of-the-art LLMs. The hierarchy of performance was clear:

  • Top Performer: GPT-5 led the pack with a 77.39% pass@1 and a staggering 91.30% pass@3.
  • Strong Contenders: Claude-3.5-Opus and Gemini-3-Pro followed closely, hovering around the 66-67% mark for pass@1.
  • The Precision of the Framework: Manual validation of the results showed that the automated JPF-based framework achieved 92% precision, proving that model checking is a highly reliable way to benchmark non-deterministic code.

Limitations and Future Work

  • State-Space Explosion: Model checking is computationally expensive. To keep the benchmark runnable, the authors had to bound the number of threads and iterations. This means some extremely subtle bugs might still slip through if they require very deep execution paths.
  • Livelocks and High-Level Logic: While JPF catches deadlocks and races, it doesn't automatically catch "livelocks" (where threads are active but making no progress) or higher-level semantic errors unless specific assertions are written into the code.
  • Language Support: Currently, CONCUR is limited to Java. The authors plan to expand the framework to other languages with strong concurrency models, like C++ or Rust, in future iterations.

Summary updated

Abstract

Leveraging Large Language Models (LLMs) for code generation has increasingly emerged as a common practice in the domain of software engineering. Relevant benchmarks have been established to evaluate the code generation capabilities of LLMs. However, existing benchmarks focus primarily on sequential code, lacking the ability to effectively evaluate LLMs on concurrent code generation. Compared to sequential code, concurrent code exhibits greater complexity and possesses unique types of bugs, such as deadlocks and race conditions, that do not occur in sequential code. Therefore, a benchmark for evaluating sequential code generation cannot be useful for evaluating concurrent code generation with LLMs. To address this gap, we designed a benchmark CONCUR specifically aimed at evaluating the capability of LLMs to generate concurrent code. CONCUR consists of 43 curated concurrency problems and leverages formal methods techniques, namely model checking, to assess the correctness of the generated code. We conducted an evaluation of a range of LLMs on CONCUR, highlighting limitations of current models. Overall, our work provides a novel direction for evaluating the capability of LLMs to generate code with focus on concurrency.
BibTeX
@misc{huang2026concur,
  author = {Jue Huang and Tarek Mahmud and Corina S. Pasareanu and Guowei Yang},
  title = {{CONCUR: Benchmarking LLMs for Concurrent Code Generation}},
  year = {2026},
  eprint = {2603.03683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.03683},
}
Figure