TL;DR
This paper reveals a subtle but potent vulnerability in self-evolving AI agents: reference-feedback poisoning. By corrupting the ground-truth answers used during an agent's "skill optimization" phase, an attacker can trick the agent into writing its own persistent backdoors—natural language rules that trigger malicious behavior—without ever touching the model's weights or code.
What Problem Does This Paper Address?
We are entering an era of "self-evolving" agents—systems like SkillOpt that improve over time by performing tasks, reflecting on their failures, and writing natural-language "Skills" (reusable instructions) to guide future behavior.
The authors identify a critical, overlooked attack vector: the supervision channel. These agents rely on reference answers to know if they succeeded or failed. If an attacker can poison even a small fraction of these references (e.g., in a shared benchmark or a scraped dataset), they can influence the agent's self-reflection process. The core question is: Can an agent be manipulated into "learning" a backdoor rule and persisting it in its own instruction set?
Methods
The researchers studied this through the lens of SkillOpt, using GPT-5.6 Luna as the core engine. They introduced a framework called Promotion-Stage Analysis to track the "life cycle" of a backdoor rule. Instead of just looking at final success, they traced whether a poisoned rule could:
- Be recognized during reflection.
- Be proposed as an executable edit.
- Survive merging and ranking against other edits.
- Be selected for a candidate Skill based on validation scores.
- Be retained in the final, persistent Skill file.
- Activate on unseen, triggered inputs.
They also used diagnostic interventions—manually relaxing certain optimizer constraints (like edit budgets)—to pinpoint exactly where attacks fail in more complex environments.
Datasets
The study utilized two primary environments:
- SearchQA: An open-domain question-answering task where the agent must provide short answers based on search snippets. This represents a "simple" text-space.
- SpreadsheetBench: A complex, tool-mediated task requiring the agent to edit Excel workbooks to satisfy instructions. This represents a "structured" space.
For evaluation, they used "frozen" held-out sets containing clean inputs, inputs with the trained trigger, and inputs with "in-family" but unseen triggers to test the scope of the backdoor.
Key Insights and Ideas
- Exposure is not Persistence: Simply seeing a poisoned example isn't enough. A rule must survive a "gauntlet" of optimization gates. In complex tasks like SpreadsheetBench, the natural attack often failed because the optimizer's "Analyst" would abstract away the specific malicious literals the attacker needed.
- The External Carrier Effect: This is a particularly devious finding. A backdoor edit that is "validation-neutral" (it doesn't help or hurt clean performance) can "hitch a ride" into the final Skill if it is bundled with a highly effective, legitimate clean edit. The optimizer accepts the whole "package" because the net gain is positive.
- Validation Poisoning as a Catalyst: While poisoning training data helps form the rule, poisoning a small fraction of the validation set is what ensures the rule is selected and retained, as it makes the backdoored candidate appear superior to the clean incumbent.
- The "Slow-Update" Shield: In some configurations, once a rule is written into a specific section of the Skill file (the "slow-update" block), it becomes shielded from further editing by the step-level Analyst, making the backdoor effectively permanent.
Key Results
The vulnerability varies significantly by task complexity:
- SearchQA (High Vulnerability): The attack was 100% successful across all tested seeds. The final Skills fired the payload on 200/200 triggered questions while maintaining near-perfect clean utility (169/200 vs 171/200 for a benign run).
- SpreadsheetBench (Resilience through Abstraction): Natural attacks failed to retain exact rules. However, when the authors used a diagnostic to allow "literal preservation," the attack succeeded, showing that the bottleneck was the optimizer's own tendency to generalize.
Limitations and Future Work
The study was limited to a single optimizer (SkillOpt) and one closed-source model (GPT-5.6 Luna). The authors note that different model sizes and optimization strategies might yield different results.
Future research directions include:
- Defensive Design: Developing "per-edit attribution" and "leave-one-edit-out" validation to detect when a malicious rule is hitchhiking on clean gains.
- Broader Modalities: Testing these attacks in multi-modal agents or those with different memory structures.
- Provenance Tracking: Investigating how to verify the integrity of reference answers in large-scale, externally maintained datasets.
Summary updated