TL;DR
SELFOP is an automated optimization algorithm that enables frozen LLM-based security agents to systematically improve their own task context (instructions, skills, and reference documents) using a chain-rule-inspired textual gradient descent. By learning from its own execution trajectories and vulnerability metadata, a smaller model optimized with SELFOP can successfully self-improve enough to outperform a much larger, more expensive frontier model baseline.
What Problem Does This Paper Address?
Large Language Model (LLM) agents are increasingly deployed for critical cybersecurity tasks like vulnerability discovery, exploit reproduction, and automated patching. However, when an agent underperforms in a new domain, improving its capabilities is notoriously difficult.
Traditional machine learning optimization levers fall short here:
- Model Fine-Tuning is computationally prohibitive and requires massive datasets of expert demonstrations or reliable reinforcement learning rewards, both of which are incredibly scarce in security.
- Harness Engineering (modifying the agent's tools and orchestration logic) requires deep, manual domain expertise and scales poorly.
- Context Optimization (prompt engineering) is the most accessible lever, but existing automated methods rely on unguided "propose-and-evaluate" loops (essentially trial-and-error), require a stronger teacher model, or depend on expensive held-out validation sets to know when to stop.
Furthermore, security tasks lack a single canonical "ground truth"—a vulnerability can be triggered by many different payloads, making standard optimization metrics unreliable and prone to shortcut learning.
Methods
The authors introduce SELFOP, a framework that treats context optimization as a discrete, text-based analogue to gradient descent.
The algorithm operates in four distinct stages:
- Chain-Rule Gradient Computation: Instead of analyzing a failure in a single shot, SELFOP uses a chain-rule analogy to backpropagate error signals through three layers: from the final binary loss ($L$) to execution scores ($s$), through the multi-step agent trajectory ($\tau$), and finally to the context artifacts ($\theta$). Crucially, it leverages available vulnerability metadata (like bug descriptions and sanitizer traces) to reconstruct a "structural invariant" of the bug before looking at the current prompt, preventing the optimizer from being biased by what the agent already did.
- Gradient Accumulation: To handle noise across a batch of tasks, SELFOP extracts strengths and weaknesses, clustering them into cross-task "themes." It ranks them by consensus and filters out low-support contradictions.
- Update Calibration (Hard vs. Soft Gradients): Surviving themes are annotated as either a hard gradient (a complete gap in the current instructions requiring new content) or a soft gradient (a rule that is already present but wasn't followed properly, requiring rephrasing or reinforcement). This prevents the prompt from bloating endlessly.
- Validation-Free Convergence Detection: Rather than running expensive forward passes on a validation set to decide when to stop, SELFOP monitors the novelty of the textual gradients themselves. When the ratio of hard-to-soft gradients drops below an empirically calibrated noise floor, the algorithm automatically stops.
Datasets
The authors evaluated SELFOP on CyberGym, a benchmark consisting of real-world vulnerability reproduction tasks sourced directly from OSS-Fuzz. CyberGym provides realistic complexity across diverse codebases, languages, and build systems, alongside crucial metadata (developer patch commits, sanitizer stack traces, and bug reports) used by SELFOP for self-supervision.
Key Insights and Ideas
- The Power of "Skill-Blind" Analysis: By forcing the optimizer to evaluate the agent's trajectory against the vulnerability metadata before reading the current instructions, the algorithm generates objective behavioral feedback rather than surface-level pattern matching.
- Textual Gradients Have "Direction" and "Magnitude": Categorizing feedback into cross-task consensus themes provides the direction for the update, while the hard/soft classification acts as a learning rate/magnitude control, preventing prompt collapse and undirected complexity growth.
- Gradients Signal Their Own End: You don't need a validation set to know when an LLM prompt optimizer is done. When the optimizer stops generating genuinely new instruction gaps (hard gradients) and only suggests tweaks to existing ones (soft gradients), the learning signal is exhausted.
Key Results
- Beating the Frontier: With fewer than 200 training examples, SELFOP drove a 17-percentage-point improvement for GPT-5.4-mini (from 38% to 55% success in generating working Proof-of-Concepts). This allowed the optimized mini model to outperform the baseline frontier GPT-5.4 model (49%), despite the frontier model being 3.5× more expensive.
- Scaling to Large Models: When applied directly to the frontier GPT-5.4 model, SELFOP achieved an 18.5-point self-improvement, lifting its success rate to 67.5%.
- True Knowledge Transfer: Skills optimized on one model successfully transferred to others. For instance, the skill optimized by GPT-5.4-mini lifted the baseline GPT-5.4 model to 62%, proving that SELFOP captures generalizable task strategies rather than model-specific phrasing quirks.
Limitations and Future Work
While highly effective, the authors note a few constraints. Due to the high API costs of running full agentic security pipelines, the evaluation was restricted to roughly 23% of the CyberGym dataset. Additionally, the learnable parameter space was scoped strictly to text-only agent skills (SKILL.md files and reference documents).
Future work includes expanding SELFOP to optimize broader context components—such as sub-agent specifications, executable verification scripts, and orchestration harness configurations—as well as testing it on adjacent security tasks like automated patch generation and threat intelligence.
Summary updated