Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

S M Abdulla Al Mamun, Akif Islam, Mohd. Ruhul AmeenarXiv 2026 · arXiv

TL;DR

Scaling up the number of LLM "specialist" agents doesn't necessarily make software safer; in fact, it can increase costs 50-fold without reducing the risk of dismissing real bugs. This study finds that while verifier feedback helps models, it also introduces new errors, suggesting that simple deterministic "gates" are still more reliable than complex multi-agent LLM architectures for software assurance.

What Problem Does This Paper Address?

In the world of software assurance, the goal is to find and fix vulnerabilities. Recently, researchers have started using Large Language Models (LLMs) to triage warnings from static analysis tools—essentially acting as a filter to decide if a reported bug is a "real" issue or a "false positive."

However, a critical problem exists: in security, the most expensive mistake isn't a missed bug, but a false clearance—wrongly labeling a real vulnerability as benign. While some believe that "more agents are all you need" (scaling up specialist roles like "data-flow analyst" or "security auditor"), this paper investigates whether this complexity actually adds value when we already have deterministic tools (like formal verifiers) that provide hard evidence.

Methods

The researchers conducted five experiments (E1–E5) to test the limits of LLM agents using the Qwen2.5-Coder-7B-Instruct model. Their approach involved:

  • Specialist Contexts: They compared a single general analyst against "teams" of up to 50 specialist roles. These roles didn't talk to each other; instead, they provided independent assessments that were aggregated via majority vote.
  • The Scope Gate: A key contribution is a deterministic post-processor (the "gate"). If a model suggests clearing a warning, the gate checks if that clearance is supported by the actual tool evidence (e.g., matching code hashes and verifier success).
  • Verifier Feedback: They tested how models perform when given raw output from JBMC (a bounded model checker for Java).

The overall pipeline is illustrated in Figure 1, showing how task data flows through specialist contexts and a majority vote before hitting the deterministic scope gate.

Overview of the experimental interfaces.

Datasets

The study utilized several well-known benchmarks and custom controls:

  • Juliet Java 1.3: Used for initial development and auditing.
  • OWASP Benchmark 1.2: A standard for testing the accuracy of vulnerability detection tools.
  • SV-COMP: A collection of formal verification tasks.
  • Defects4J: A database of real-world Java bugs.
  • Authored Controls: Eight custom-written programs with known null-pointer exceptions (NPEs) to test scaling limits.

Key Insights and Ideas

  • Scaling has diminishing (or negative) returns: Increasing the number of agents from 1 to 50 increased token usage by 49x but did not improve the model's ability to catch unsafe clearances.
  • "Applicability Confusion": One of the most surprising findings was that models often rejected valid evidence of a bug. When shown a counterexample (proof that a bug exists), the model would sometimes judge the program as buggy and therefore conclude the evidence was wrong, rather than accepting it as proof of the defect.
  • Feedback is a double-edged sword: Providing raw verifier feedback helped the model reach more correct verdicts, but it also significantly increased the number of incorrect verdicts (hallucinations or misinterpretations).
  • Deterministic tools are still king: In almost every test, a simple deterministic check (the "gate") was more effective at preventing false clearances than any configuration of LLM agents.

Key Results

The quantitative results paint a sobering picture for LLM-only assurance:

  • Cost vs. Safety: As shown in Figure 2, even with a massive budget increase (49x more tokens), the "unsafe clearance" rate remained near 100% for the ungated models. Only the deterministic gate held it at zero.
Fig. 2. E4: unsafe clearance (ungated, mean of 3 seeds; whiskers show the seed range) against total tokens. A 49× budget increase leaves unsafe clearance near 100%; the deterministic gate holds it at zero for every 𝑘.
  • Feedback Impact: In the SV-COMP tests (Figure 3), raw feedback increased correct verdicts from 12.7 to 22.0, but incorrect verdicts also jumped from 1 to 7. A simple deterministic policy (JBMC + JVM) outperformed the model entirely with 26 correct verdicts.
Fig. 3. E3 verdicts on 32 frozen tasks (means of three seeds). Raw feedback raises both correct and incorrect counts. The deterministic comparator additionally uses JVM validation.
  • Evidence Rejection: In the misleading-evidence challenge (Figure 4), models rejected 70–100% of valid evidence packages. This highlights a fundamental struggle LLMs have with recognizing formal proof when the underlying code is messy.
Fig. 4. E5 misleading-evidence challenge (100 corrupted and 20 clean packages per seed). The gate removes erroneous acceptance but cannot repair the model’s rejection of valid evidence; the scope validator alone makes neither error.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Size: They only tested relatively small models (7B and 1.5B parameters). Larger models might exhibit different reasoning patterns.
  • Synthetic Data: Benchmarks like OWASP are synthetic; real-world, large-scale software might present different challenges.
  • Static Roles: The study used a fixed catalogue of roles and a simple majority vote. More dynamic agent architectures (with iterative communication) might perform better.

Future Work: The authors suggest that instead of just adding more agents, research should focus on "budget-matched" evaluations—comparing a single model with a large inference budget against a multi-agent system with the same total cost. They also emphasize the need for better "applicability reasoning" to help models understand when formal evidence is valid.

Summary updated

Abstract

Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.
BibTeX
@misc{mamun2026beyond,
  author = {S M Abdulla Al Mamun and Akif Islam and Mohd. Ruhul Ameen},
  title = {{Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance}},
  year = {2026},
  eprint = {2610.36947},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36947},
}
Figure