TL;DR
This paper demonstrates that local, open-weight LLMs (specifically the Qwen family) are now capable of autonomously reproducing software vulnerabilities on a single consumer GPU. The researchers found that while most models can correctly identify a bug's location and cause, the primary bottleneck is the "last mile": translating that conceptual understanding into a byte-exact crashing input.
What Problem Does This Paper Address?
Vulnerability reproduction—the process of turning a bug description into a working Proof-of-Concept (PoC) exploit—is a critical task for security researchers and developers. While frontier models like GPT-4 have shown promise here, using them presents significant hurdles for security professionals:
- Privacy: Sending unpatched, proprietary code to a third-party API is often a non-starter for labs.
- Control: API users cannot control model parameters, quantization levels, or underlying hardware.
- Reliability: Rate limits and service downtime can derail automated security pipelines.
The authors investigate whether smaller, "open-weight" models (which can be run locally) have matured enough to handle the complex, multi-step "agentic" workflows required to find and trigger crashes in real-world software.
Methods
The researchers evaluated twelve configurations of the Qwen model family, varying across four dimensions:
- Generations: Qwen 3.5, 3.6, and 3.8.
- Parameter Counts: 4B, 9B, 27B, and a 35B Mixture-of-Experts (MoE) model.
- Architecture: Dense vs. Mixture-of-Experts.
- Quantization: BF16 (16-bit), FP8 (8-bit), and INT4 (4-bit).
The models acted as agents using the pi coding agent harness. The agent operates in a closed self-verification loop: it reads the source code and bug description, crafts a candidate PoC, and submits it to an execution server. The server provides feedback (exit codes and output), allowing the agent to refine its input until it successfully crashes the vulnerable build but not the patched one.
All experiments were conducted on a single NVIDIA RTX PRO 6000 Blackwell (96 GB), representing a high-end but accessible consumer/workstation setup.
Datasets
The study utilized CyberGym, a benchmark for real-world cybersecurity tasks. The authors selected a representative subset of 10 tasks involving projects like GraphicsMagick, FFmpeg, and binutils.
- Difficulty Level: The agents were tested at "Level 1," where they receive the source tree and a natural language description of the bug, but no fuzzer output or upstream fixes.
- Task Split: The tasks were categorized into six "easier" tasks (solved by most models) and four "discriminating" tasks (which separated the high-performing models from the rest).
Key Insights and Ideas
- Analysis vs. Artefact: A striking finding was that almost every model—even those that failed—correctly identified the vulnerable function and the logic error within the first few messages. The failure wasn't a lack of "understanding," but rather an inability to generate the precise binary or text structure required to reach that vulnerable code path.
- The 27B Threshold: There is a clear capability floor. Models smaller than 27B parameters failed to solve any of the "discriminating" (harder) tasks.
- Quantization as a Speed Knob: Contrary to some fears that quantization destroys reasoning, the researchers found that 4-bit and 8-bit versions often performed as well as (or better than) their 16-bit counterparts. This is largely because higher throughput allowed the agents to make more attempts within the 7200-second time limit.
- MoE Efficiency: The Mixture-of-Experts (MoE) architecture proved highly effective. It reached the same success rates as dense models but in significantly less wall-clock time, making it the superior choice for time-constrained deployments.
Key Results
- Overall Success: The twelve configurations solved 211 out of 360 total runs.
- Top Performer: The Qwen 3.8 27B (FP8) was the most capable, solving 8 out of 12 runs on the hardest tasks.
- Complexity Correlation: There was a strong negative correlation (-0.81) between the size of the reference PoC and the success rate. If a crash requires a 250 KB input file, models struggle significantly more than if the crash is triggered by a 10-byte file.
- Cost of Failure: Successful runs were surprisingly efficient, with a median of 1.0 million tokens. However, failing runs consumed a median of 7.7 million tokens as the models entered "resampling loops," repeatedly trying similar but incorrect inputs.
Limitations and Future Work
- Model Diversity: The study was limited to the Qwen family. Testing other open-weight families (like Llama or Mistral) would confirm if these thresholds are universal.
- Sample Size: Due to the high compute cost (100 hours for the sweep), the study used a small subset of 10 tasks. A broader sweep could provide more statistical resolution.
- Agent Feedback: The authors suggest that the next major leap in performance might come from improving the feedback the agent receives from the environment (e.g., better error localization) rather than just increasing model size.
- Failure Mode Classification: Future work could systematically classify why artefacts fail (e.g., wrong magic bytes, incorrect offsets) to help build better defensive coding habits in agents.
Summary updated