TL;DR
While Large Language Models (LLMs) have become surprisingly good at writing sequential code, they have a massive blind spot: concurrency. This paper introduces CONCUR, the first benchmark specifically designed to evaluate LLMs on multi-threaded programming using formal model checking to catch non-deterministic bugs like deadlocks and race conditions that standard tests miss.
What Problem Does This Paper Address?
Writing code that does one thing at a time is relatively straightforward. Writing code where multiple threads interact simultaneously is a nightmare. Concurrent programming introduces "non-determinism"—the idea that a program might work perfectly 99 times but crash on the 100th because of a specific, rare interleaving of tasks.
Existing benchmarks like HumanEval or MBPP focus almost exclusively on sequential logic. They rely on unit tests or static similarity metrics (like CodeBLEU) to judge correctness. However, a concurrent program can pass a unit test simply because the "bad" thread schedule didn't happen during that specific run. Furthermore, static metrics only check if the code looks right, not if it behaves right under pressure. The authors argue that evaluating LLMs only on sequential code leaves a critical gap in our understanding of their true software engineering capabilities.
Methods
The researchers moved beyond simple "run and see" testing by employing Formal Methods. Specifically, they used a tool called Java PathFinder (JPF), an explicit-state model checker.
Unlike a standard JVM that just executes code once, JPF systematically explores all possible thread interleavings and execution paths within a defined bound. This allows the benchmark to rigorously detect:
- Deadlocks: Threads waiting on each other forever.
- Race Conditions: Multiple threads accessing shared data without proper synchronization.
- Starvation: High-priority threads hogging resources.
- Uncaught Exceptions: Errors that only trigger under specific concurrent conditions.
The evaluation framework is a two-stage process: first, it checks if the code compiles in a Java 8 environment; then, it hands the successful builds to JPF for deep behavioral verification.
Datasets
The authors introduced the CONCUR benchmark, which consists of 115 concurrency problems:
- 43 Base Problems: Curated from the classic textbook Java Concurrency in Practice (Goetz, 2006). These cover fundamental constructs like
synchronizedblocks,ReentrantLock,Semaphore, andExecutorService. - 72 Mutant Variants: To prevent models from simply "memorizing" textbook solutions, the authors used Gemini to generate variations of the base problems, which were then manually validated for semantic correctness.
All problems are paired with structured prompts and validated ground-truth implementations in Java.
Key Insights and Ideas
- The "Conceptual vs. Implementation" Gap: One of the most interesting findings was the "Single Thread" error. LLMs often wrote perfectly thread-safe classes (like a thread-safe counter) but then wrote a
mainmethod that only used a single thread. They "understood" the concept of thread safety but failed to implement the actual concurrency required by the prompt. - CodeBLEU is Unreliable for Concurrency: The study found that CodeBLEU scores (which measure how much the generated code "looks like" the reference code) correlate very poorly with actual correctness. A model could have a high CodeBLEU score while containing a fatal deadlock that JPF easily caught.
- The Power of Multiple Attempts: Mid-performing models saw a massive boost when moving from
pass@1(one attempt) topass@3(three attempts). For example,codeqwen:7bmore than doubled its accuracy. This suggests that while these models "know" the right patterns, they are inconsistent in applying them. - Syntax is Still a Hurdle: Even in 2026, syntax errors remain the leading cause of failure (71% of compilation errors), often because LLMs try to use third-party libraries despite being told to use only standard Java 8.
Key Results
The researchers evaluated 23 state-of-the-art LLMs. The hierarchy of performance was clear:
- Top Performer: GPT-5 led the pack with a 77.39% pass@1 and a staggering 91.30% pass@3.
- Strong Contenders: Claude-3.5-Opus and Gemini-3-Pro followed closely, hovering around the 66-67% mark for pass@1.
- The Precision of the Framework: Manual validation of the results showed that the automated JPF-based framework achieved 92% precision, proving that model checking is a highly reliable way to benchmark non-deterministic code.
Limitations and Future Work
- State-Space Explosion: Model checking is computationally expensive. To keep the benchmark runnable, the authors had to bound the number of threads and iterations. This means some extremely subtle bugs might still slip through if they require very deep execution paths.
- Livelocks and High-Level Logic: While JPF catches deadlocks and races, it doesn't automatically catch "livelocks" (where threads are active but making no progress) or higher-level semantic errors unless specific assertions are written into the code.
- Language Support: Currently, CONCUR is limited to Java. The authors plan to expand the framework to other languages with strong concurrency models, like C++ or Rust, in future iterations.
Summary updated