Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment

Ziqian Ban, Haoxiang Yang, Yanfei Zhang, Shixin Liang, Haoran Yu, Xu Lin, Shaobin Zhao, Xuran Li, Zhiyi ChengarXiv 2026 · arXiv

TL;DR

Just because an AI agent's score goes up doesn't mean it's actually getting smarter. This paper introduces a rigorous "Artifact-to-Capability" diagnostic framework to prove that standard performance metrics often mask fundamental failures in how self-evolving agents learn and reuse skills.

What Problem Does This Paper Address?

We are currently in an era of "self-evolving" agents—AI systems that learn from their own experiences by creating "artifacts" like textual reflections, code snippets, or skill libraries. Usually, researchers claim these agents are successful if they see three "proxy signals": they generate more artifacts, their behavior changes when those artifacts are loaded, and their overall task scores improve.

The authors argue that these signals are deceptive. A higher score might come from an agent simply learning to follow basic rules (like not playing on an occupied square) rather than actually getting better at the core task (like winning the game). To fix this, the paper defines what it actually means for an artifact to become a capability and provides a way to "microscope" the evolution process to find exactly where it breaks.

Methods

The researchers propose the Artifact-to-Capability framework, which decomposes capability formation into a "capability chain" of five sequentially dependent properties:

  1. Artifact formation: A persistent structure (like a code file) is created.
  2. Behavioral activation: The artifact actually changes the agent's actions.
  3. Task-relation realization: The change is positive and advances the task objective.
  4. Boundary-safe reuse: The new skill doesn't break existing ones when combined.
  5. Capability extension: Later versions of the agent build upon and preserve these gains.
Figure 2: The self-evolution loop: experience, evaluation, reflection, artifact updates, and feedback.

They test this using freestyle Gomoku (Five-in-a-Row) as a controlled environment. Gomoku is ideal because it has simple, absolute rules, yet requires complex spatial reasoning that is difficult for LLMs (see Figure 1). Crucially, it allows for "replayable states," meaning researchers can test an agent with and without a specific memory on the exact same board position to see what changed.

Datasets

The study uses freestyle Gomoku with a 15x15 board. Evaluation is performed on 1,876 fixed board states:

  • Set A: 120 manually constructed tactical states (e.g., immediate win opportunities).
  • Set B: 1,756 states drawn from actual games played during the evolution of one of the agents.

The agents are tested against three programmatic opponents of increasing difficulty: rule-based, threat-aware, and search_2.

Key Insights and Ideas

The "aha moment" of the paper is the discovery of three distinct types of "capability-formation breaks" in popular agent frameworks:

  • Legalization Gain (Reflexion): The agent's score rose, but only because it learned to stop making illegal moves. When looking only at its legal move choices, the agent actually performed worse after "reflecting" on its mistakes.
  • Compositional Override (Voyager): Individual skills worked fine in isolation, but the "calling script" that managed them was flawed. In "conflict states"—where both the agent and the opponent could win in one move—the agent's defensive skill would trigger and block the opponent, completely ignoring its own immediate chance to win the game.
  • Loss-Only Drift (DGM): While the agent initially formed a capability, later generations of the "evolving" code stopped adding new skills and actually began losing the ones they had already mastered, despite the internal evolutionary scores remaining high.
Figure 4: Voyager against search_2. Twenty independent games with different random seeds converge to four move sequences (G1–G4); numbers give move order. In every sequence, Voyager forms an open four whose two immediate winning points (blue stars) stay empty. At move 19, a conflict state, auto_player returns early on the defense branch and blocks one of search_2’s immediate winning points (orange ring); search_2 completes five at the other on move 20 (orange line). Immediate winning points are recomputed from the rules; Appendix E.1 lists them and the auto_player call order.

Key Results

The authors diagnosed three well-known self-evolution mechanisms and found that while all showed "positive" signals, all failed the capability chain:

  • Reflexion: While the overall pass rate rose, the legal pass rate (the quality of legal moves) dropped by 1.6–4.4 percentage points on the large test set.
  • Voyager: Despite forming "open fours" (a strong offensive position) in every game, it failed to complete a win in 0 out of 20 immediate-win opportunities during actual gameplay due to the compositional override.
  • DGM: After reaching a peak version (OB-3), subsequent versions (OB-5, OB-6) added zero newly passing states and only lost existing ones, a phenomenon the authors call "loss-only drift."
Figure 5: Reflexion on set B: change in each rate relative to empty memory, one dot per frozen memory (six in total; each memory keeps the same horizontal position in all three columns). All six raise the overall pass rate and the legality rate, yet all six lower the legal pass rate. The empty-memory rerun changes the three rates by +2.5%, +2.4%, and +0.6%. Full results for both sets are in Table C.1.

Limitations and Future Work

The primary limitation is the environment: freestyle Gomoku is a discrete, rule-governed game. While this "microscope" is powerful for identifying logic breaks, it's unclear how these properties translate to open-ended, unstructured tasks (like writing a novel or browsing the web) where "correctness" is subjective and states are harder to replay exactly.

The authors suggest that future work should focus on operationalizing this framework for more complex, real-world agent tasks to ensure that "self-evolution" is resulting in genuine, robust capabilities rather than just metric-chasing.

Summary updated

Abstract

Self-evolving agents write interaction experience into artifacts and reuse them in later tasks. Most work supports such methods with three proxy signals: artifacts accumulate, behavior changes once they are loaded, and metrics rise. We call this signal-based evaluation. What these signals are meant to indicate, however, has not been defined. To fill this gap, we define capability formation from artifacts as five sequentially dependent properties, two of which form a checkpoint for capability formation, and introduce three-scale diagnosis, which judges these properties at the state, game, and lineage levels. In freestyle Gomoku, Reflexion, Voyager, and DGM evolve with the same base model and all show the three signals, yet each exhibits a distinct capability-formation break. Reflexion shows a legalization gain: its pass rate rises only because more of its moves are legal. Voyager shows a compositional override: components that are sound on their own combine to miss all 20 available one-move wins. DGM shows loss-only drift: after passing the checkpoint, later versions only lose passing states. These counterexamples show that signal-based evaluation cannot determine whether a capability has formed.
BibTeX
@misc{ban2026artifact,
  author = {Ziqian Ban and Haoxiang Yang and Yanfei Zhang and Shixin Liang and Haoran Yu and Xu Lin and Shaobin Zhao and Xuran Li and Zhiyi Cheng},
  title = {{Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment}},
  year = {2026},
  eprint = {2611.04952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04952},
}
Figure