When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization

Louise Nayeong Koh, Yaechan Lee, Myeongjong Lee, Doyeon Kim, Juyoung Park, Seunggyu Kim, Inseo Kim, Chaerin Jung, Hyogon RyuarXiv 2026 · arXiv

TL;DR

This paper reveals a subtle but potent vulnerability in self-evolving AI agents: reference-feedback poisoning. By corrupting the ground-truth answers used during an agent's "skill optimization" phase, an attacker can trick the agent into writing its own persistent backdoors—natural language rules that trigger malicious behavior—without ever touching the model's weights or code.

What Problem Does This Paper Address?

We are entering an era of "self-evolving" agents—systems like SkillOpt that improve over time by performing tasks, reflecting on their failures, and writing natural-language "Skills" (reusable instructions) to guide future behavior.

The authors identify a critical, overlooked attack vector: the supervision channel. These agents rely on reference answers to know if they succeeded or failed. If an attacker can poison even a small fraction of these references (e.g., in a shared benchmark or a scraped dataset), they can influence the agent's self-reflection process. The core question is: Can an agent be manipulated into "learning" a backdoor rule and persisting it in its own instruction set?

Methods

The researchers studied this through the lens of SkillOpt, using GPT-5.6 Luna as the core engine. They introduced a framework called Promotion-Stage Analysis to track the "life cycle" of a backdoor rule. Instead of just looking at final success, they traced whether a poisoned rule could:

  1. Be recognized during reflection.
  2. Be proposed as an executable edit.
  3. Survive merging and ranking against other edits.
  4. Be selected for a candidate Skill based on validation scores.
  5. Be retained in the final, persistent Skill file.
  6. Activate on unseen, triggered inputs.
Figure 1: Victim optimization loop. Execution is scored against references; reflection proposes edits; validation and selection determine which candidate is persisted. The attacker supplies only tasks and references.

They also used diagnostic interventions—manually relaxing certain optimizer constraints (like edit budgets)—to pinpoint exactly where attacks fail in more complex environments.

Datasets

The study utilized two primary environments:

  • SearchQA: An open-domain question-answering task where the agent must provide short answers based on search snippets. This represents a "simple" text-space.
  • SpreadsheetBench: A complex, tool-mediated task requiring the agent to edit Excel workbooks to satisfy instructions. This represents a "structured" space.

For evaluation, they used "frozen" held-out sets containing clean inputs, inputs with the trained trigger, and inputs with "in-family" but unseen triggers to test the scope of the backdoor.

Key Insights and Ideas

  • Exposure is not Persistence: Simply seeing a poisoned example isn't enough. A rule must survive a "gauntlet" of optimization gates. In complex tasks like SpreadsheetBench, the natural attack often failed because the optimizer's "Analyst" would abstract away the specific malicious literals the attacker needed.
  • The External Carrier Effect: This is a particularly devious finding. A backdoor edit that is "validation-neutral" (it doesn't help or hurt clean performance) can "hitch a ride" into the final Skill if it is bundled with a highly effective, legitimate clean edit. The optimizer accepts the whole "package" because the net gain is positive.
  • Validation Poisoning as a Catalyst: While poisoning training data helps form the rule, poisoning a small fraction of the validation set is what ensures the rule is selected and retained, as it makes the backdoored candidate appear superior to the clean incumbent.
  • The "Slow-Update" Shield: In some configurations, once a rule is written into a specific section of the Skill file (the "slow-update" block), it becomes shielded from further editing by the step-level Analyst, making the backdoor effectively permanent.

Key Results

The vulnerability varies significantly by task complexity:

  • SearchQA (High Vulnerability): The attack was 100% successful across all tested seeds. The final Skills fired the payload on 200/200 triggered questions while maintaining near-perfect clean utility (169/200 vs 171/200 for a benign run).
  • SpreadsheetBench (Resilience through Abstraction): Natural attacks failed to retain exact rules. However, when the authors used a diagnostic to allow "literal preservation," the attack succeeded, showing that the bottleneck was the optimizer's own tendency to generalize.
Table 4: Furthest stage (Table 2) reached by an exact trigger–payload rule, per seed. “gen.” denotes a generalized exact-payload rule. Activation counts are payload emissions on held-out triggered inputs.

Limitations and Future Work

The study was limited to a single optimizer (SkillOpt) and one closed-source model (GPT-5.6 Luna). The authors note that different model sizes and optimization strategies might yield different results.

Future research directions include:

  • Defensive Design: Developing "per-edit attribution" and "leave-one-edit-out" validation to detect when a malicious rule is hitchhiking on clean gains.
  • Broader Modalities: Testing these attacks in multi-modal agents or those with different memory structures.
  • Provenance Tracking: Investigating how to verify the integrity of reference answers in large-scale, externally maintained datasets.

Summary updated

Abstract

Reference answers are a supervision channel in text-space Skill optimization, which produces reusable text artifacts called Skills that guide future agent behavior. Poisoning those answers can induce a persistent trigger-conditioned rule without giving the attacker access to the model, optimizer, or Skill file. We study this reference-feedback poisoning in SkillOpt by tracing an attacker-supplied trigger--payload mapping from reflection to held-out activation. In SearchQA, where the payload replaces a short answer, the natural attack retains an exact rule in all three seeds; each final Skill emits the payload on 200/200 triggered questions and 0/200 clean questions. In seed 42, removing the rule eliminates activation, and an in-run comparison shows similar clean accuracy to benign SkillOpt (169/200 versus 171/200). In SpreadsheetBench, where the payload requires an additional workbook edit, no natural run retains an exact rule across three seeds. Diagnostic interventions locate two successive bottlenecks: the Analyst abstracts away required literals, and, once literal preservation is enabled, edit-budget ranking can remove exact edits before candidate construction. Poisoned references can therefore create persistent backdoors, but exposure alone does not ensure that the induced rule is formed, selected, and retained.
BibTeX
@misc{koh2026when,
  author = {Louise Nayeong Koh and Yaechan Lee and Myeongjong Lee and Doyeon Kim and Juyoung Park and Seunggyu Kim and Inseo Kim and Chaerin Jung and Hyogon Ryu},
  title = {{When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization}},
  year = {2026},
  eprint = {2611.09348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09348},
}
Figure