Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks

Jorge García-Carrasco, Javier Sanchis, Álvaro Navarro, Alejandro Maté, Juan TrujilloarXiv 2026 · arXiv

TL;DR

This paper demonstrates that local, open-weight LLMs (specifically the Qwen family) are now capable of autonomously reproducing software vulnerabilities on a single consumer GPU. The researchers found that while most models can correctly identify a bug's location and cause, the primary bottleneck is the "last mile": translating that conceptual understanding into a byte-exact crashing input.

What Problem Does This Paper Address?

Vulnerability reproduction—the process of turning a bug description into a working Proof-of-Concept (PoC) exploit—is a critical task for security researchers and developers. While frontier models like GPT-4 have shown promise here, using them presents significant hurdles for security professionals:

  1. Privacy: Sending unpatched, proprietary code to a third-party API is often a non-starter for labs.
  2. Control: API users cannot control model parameters, quantization levels, or underlying hardware.
  3. Reliability: Rate limits and service downtime can derail automated security pipelines.

The authors investigate whether smaller, "open-weight" models (which can be run locally) have matured enough to handle the complex, multi-step "agentic" workflows required to find and trigger crashes in real-world software.

Methods

The researchers evaluated twelve configurations of the Qwen model family, varying across four dimensions:

  • Generations: Qwen 3.5, 3.6, and 3.8.
  • Parameter Counts: 4B, 9B, 27B, and a 35B Mixture-of-Experts (MoE) model.
  • Architecture: Dense vs. Mixture-of-Experts.
  • Quantization: BF16 (16-bit), FP8 (8-bit), and INT4 (4-bit).

The models acted as agents using the pi coding agent harness. The agent operates in a closed self-verification loop: it reads the source code and bug description, crafts a candidate PoC, and submits it to an execution server. The server provides feedback (exit codes and output), allowing the agent to refine its input until it successfully crashes the vulnerable build but not the patched one.

Figure 1: Overview of the evaluation loop.

All experiments were conducted on a single NVIDIA RTX PRO 6000 Blackwell (96 GB), representing a high-end but accessible consumer/workstation setup.

Datasets

The study utilized CyberGym, a benchmark for real-world cybersecurity tasks. The authors selected a representative subset of 10 tasks involving projects like GraphicsMagick, FFmpeg, and binutils.

  • Difficulty Level: The agents were tested at "Level 1," where they receive the source tree and a natural language description of the bug, but no fuzzer output or upstream fixes.
  • Task Split: The tasks were categorized into six "easier" tasks (solved by most models) and four "discriminating" tasks (which separated the high-performing models from the rest).

Key Insights and Ideas

  • Analysis vs. Artefact: A striking finding was that almost every model—even those that failed—correctly identified the vulnerable function and the logic error within the first few messages. The failure wasn't a lack of "understanding," but rather an inability to generate the precise binary or text structure required to reach that vulnerable code path.
  • The 27B Threshold: There is a clear capability floor. Models smaller than 27B parameters failed to solve any of the "discriminating" (harder) tasks.
  • Quantization as a Speed Knob: Contrary to some fears that quantization destroys reasoning, the researchers found that 4-bit and 8-bit versions often performed as well as (or better than) their 16-bit counterparts. This is largely because higher throughput allowed the agents to make more attempts within the 7200-second time limit.
  • MoE Efficiency: The Mixture-of-Experts (MoE) architecture proved highly effective. It reached the same success rates as dense models but in significantly less wall-clock time, making it the superior choice for time-constrained deployments.

Key Results

  • Overall Success: The twelve configurations solved 211 out of 360 total runs.
  • Top Performer: The Qwen 3.8 27B (FP8) was the most capable, solving 8 out of 12 runs on the hardest tasks.
  • Complexity Correlation: There was a strong negative correlation (-0.81) between the size of the reference PoC and the success rate. If a crash requires a 250 KB input file, models struggle significantly more than if the crash is triggered by a 10-byte file.
  • Cost of Failure: Successful runs were surprisingly efficient, with a median of 1.0 million tokens. However, failing runs consumed a median of 7.7 million tokens as the models entered "resampling loops," repeatedly trying similar but incorrect inputs.

Limitations and Future Work

  • Model Diversity: The study was limited to the Qwen family. Testing other open-weight families (like Llama or Mistral) would confirm if these thresholds are universal.
  • Sample Size: Due to the high compute cost (100 hours for the sweep), the study used a small subset of 10 tasks. A broader sweep could provide more statistical resolution.
  • Agent Feedback: The authors suggest that the next major leap in performance might come from improving the feedback the agent receives from the environment (e.g., better error localization) rather than just increasing model size.
  • Failure Mode Classification: Future work could systematically classify why artefacts fail (e.g., wrong magic bytes, incorrect offsets) to help build better defensive coding habits in agents.

Summary updated

Abstract

Open-weight language models that fit on a single consumer GPU are becoming capable on a growing range of tasks, and they matter because they can run locally without sending the code to a hosted service. Vulnerability reproduction, turning a vulnerability description into an input that crashes the vulnerable build, is a natural target for them, since it is an automated analysis task closer to test generation than to code comprehension and it runs against a build that must be executed rather than read. In order to assess whether current models can perform it, we evaluate twelve configurations of local models, spanning three generations, two architectures and three quantization levels, on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness, and release the full artefact. We show that every configuration, down to the 4B model, solves most of the six easier tasks, and configurations separate only on the remaining four. For the harder tasks, the 27B model is required. Quantization did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture of experts reached the same scores in less wall clock. We also examine the agent trajectories. Almost every run we examined, including those that fail, names the vulnerable function and states the correct hypothesis, and what separates a solved task from an unsolved one is whether that hypothesis becomes a working input.
BibTeX
@misc{garciacarrasco2026analysis,
  author = {Jorge García-Carrasco and Javier Sanchis and Álvaro Navarro and Alejandro Maté and Juan Trujillo},
  title = {{Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks}},
  year = {2026},
  eprint = {2611.06257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06257},
}
Figure