TL;DR
Vulnerability databases like NVD often have "blind spots" where older, unsupported software versions are incorrectly assumed to be safe. CrossPloit is a new benchmark of 127 migration instances that challenges LLM agents to prove these versions are exploitable by adapting existing Proof-of-Concept (PoC) exploits. The study reveals that while top-tier models like Claude Opus-4.6 can resolve up to 58.3% of these cases, complex architectural shifts in attack vectors remain a significant hurdle for current automation.
What Problem Does This Paper Address?
Modern security relies on "affected-version" lists. If you run a version of a library that isn't listed in a CVE entry, your scanners, patching pipelines, and incident responders will likely treat it as safe. However, these lists are frequently incomplete, especially for older branches that have reached "end-of-life" but are still widely deployed.
Proving a version is vulnerable requires executable evidence: a working PoC. But you can't just "replay" an exploit from a newer version. APIs get renamed, request paths change, and the environment (like Python or database versions) might be fundamentally different. This task, called cross-version PoC migration, is a complex puzzle involving source-code analysis, environment repair, and execution-based verification.
Methods
The authors developed a rigorous pipeline to build and test the CrossPloit benchmark:
- PoC Generation: They reconstructed 97 reference PoCs for Python web CVEs from scratch, ensuring each had a deterministic, vulnerability-specific "oracle" (a way to programmatically check if the exploit worked).
- Manual Migration: Human experts attempted to migrate these PoCs to five prior versions not listed as affected. This resulted in 127 validated migration instances across 45 CVEs.
- Agent Evaluation: They tested 18 different LLM agent configurations using the OpenHands and SWE-agent harnesses, paired with nine state-of-the-art models (e.g., GPT-5.3, Claude Opus-4.6).
- Baseline Comparison: They adapted VulScope, a trace-guided migration tool originally for C/C++, to the Python web setting (VulScope-Web) to see how traditional input-mutation techniques compare to LLMs.
Datasets
The primary contribution is the CrossPloit benchmark:
- Scope: 127 migration instances across 45 CVEs in the Python web ecosystem (Django, Flask, FastAPI, Web2py, etc.).
- Components: Each instance includes a version-pinned Docker container, a reference PoC, and a vulnerability-specific oracle.
- Diversity: The benchmark covers various vulnerability types, dominated by SQL injection (CWE-89), XSS (CWE-79), and Path Traversal (CWE-22).
Key Insights and Ideas
- The Massive Blind Spot: Benchmark construction confirmed 149 vulnerable prior versions that were absent from all four major vulnerability databases (NVD, OSV, GHSA, Snyk).
- Replay is Insufficient: Only about 15% of these versions could be exploited by simply re-running the original PoC. The rest required active migration (code changes).
- The "Attack Vector Shift" Wall: The hardest migrations are those where the original entry point is gone, but the bug is still reachable through a different path. No agent configuration could solve more than 37.9% of these cases.
- Context Matters: An ablation study showed that withholding the reference PoC and environment info caused the resolve rate to plummet from 50.4% to 15.0%. LLMs aren't just "finding" bugs; they are heavily reliant on the provided templates to bridge version gaps.
- Success Drops with Distance: As the version distance from the disclosed version increases, the likelihood of a successful migration (and the presence of the vulnerability itself) decreases significantly.
Key Results
- Top Performer: The best configuration was OpenHands with Claude Opus-4.6, which resolved 58.3% (74/127) of the migration instances.
- Baseline Failure: The adapted trace-guided baseline (VulScope-Web) only resolved 8.7% of instances, primarily because it can only mutate inputs and cannot fix broken APIs or environments.
- Failure Analysis: Higher-performing models were better at identifying the correct API/entry point but still often failed at the "last mile"—completing the execution and verifying it with the oracle.
- Human Effort: Constructing the benchmark was labor-intensive, requiring approximately 242 person-hours for PoC generation and 291 hours for migration attempts.
Limitations and Future Work
- Ecosystem Scope: The study is currently limited to the Python web ecosystem. The authors note that results might differ in other languages or application types.
- Evaluation Depth: Each tool was run only once per instance; repeated runs are needed to assess the stability of these resolve rates.
- Metadata Auditing: While the benchmark focuses on confirmed vulnerable versions, a real-world auditing tool would also need to be evaluated on its ability to correctly identify unaffected versions (minimizing false positives).
- Future Directions: The authors suggest extending the benchmark to other ecosystems and improving agent support for historical dependency resolution, which remains a persistent cause of failure.
Summary updated