Do LLM Agents Still Need to Be Told How to Self-Improve?

Shobhita Sundaram, Cansu Sancaktar, Phillip IsolaarXiv 2026 · arXiv

TL;DR

This paper investigates whether advanced LLM agents can autonomously organize their own self-improvement procedures without human-prescribed learning loops. The researchers found that sufficiently capable models (like Claude Opus-5 and GPT-6 Astra) can independently invent sophisticated learning strategies—such as A/B testing, difficulty filtering, and validation suites—to improve their performance on complex tasks using only self-generated experience.

What Problem Does This Paper Address?

Traditionally, machine learning practitioners design the "loop" for self-improvement: they specify how data is generated, how the model is evaluated, and how it should be updated. As LLM agents become more capable, a fundamental question emerges: Do we still need to tell them how to improve?

The authors explore a paradigm of "self-organized self-improvement." Instead of giving an agent a specific algorithm (like "generate 1,000 puzzles and fine-tune on the solutions"), they provide the agent with a general goal—"improve your ability to play games"—and a set of primitive tools. The challenge is for the agent to decide which games to write, how to measure its own progress, and which lessons to retain, all within a data-free sandbox.

Methods

The researchers used a Teacher-Player setup where two instances of the same frozen LLM interact:

  1. The Player: Plays turn-based games with undisclosed mechanics.
  2. The Teacher: Given a compute budget and a toolset {Read, Write, Bash, Gameplay}, the teacher's goal is to optimize the player's "context" (prompts and auxiliary skill files).

Crucially, the environment is a constrained sandbox. The teacher has no access to real-world data or human-written games. It must use the Write tool to create its own practice games in PuzzleScript (a 2D puzzle DSL) or Python, use the Gameplay tool to observe the player's attempts, and then Edit the player's instructions based on observed failures.

The teacher agent autonomously iterates through game generation, player observation, and context editing.

Datasets

The study evaluates self-improvement and generalization across several benchmarks:

  • PuzzleScript-t20/t30: 2D grid puzzles with 20- and 30-turn horizons.
  • ARC-AGI-3 (Level 1): Tests transfer to different 2D game engines and representations.
  • MazeBench: A 3D open-world navigation task represented via isometric ASCII projections, testing generalization to 3D mechanics and longer horizons.
Examples of the diverse evaluation benchmarks used to test the transferability of learned artifacts.

Key Insights and Ideas

  • Emergent Learning Algorithms: Capable models didn't just practice randomly. They independently implemented strategies from the machine learning literature, such as A/B testing (comparing player performance with and without a specific prompt update) and curriculum generation.
  • Cognitive Alignment: The agents' behaviors aligned with Self-Regulated Learning (SRL), a classic cognitive framework. As shown in Figure 6, higher-performing models scored significantly better across subprocesses like "Strategic Planning," "Self-Experimentation," and "Causal Attribution."
  • The "Goldilocks" Zone of Learnability: Successful agents naturally gravitated toward tasks of intermediate difficulty. They quickly discarded trivial games (100% win rate) and focused their "replay" efforts on games where the player had a 20-60% success rate, mirroring the "Zone of Proximal Development" in developmental psychology.
  • Cross-Model Transfer: Interestingly, the text-based "lessons" learned by one model (e.g., GPT-6 Astra) were often useful to another (e.g., Claude Opus-5), suggesting that the agents are discovering general principles of reasoning rather than just overfitting to their own quirks.
Radar charts showing how self-improving models align with human-recognizable self-regulated learning subprocesses.

Key Results

  • Performance Gains: The most capable models showed significant autonomous improvement. For example, Claude Opus-5 improved its win rate by +20% on held-out PuzzleScript tasks.
  • Parity with Prescribed Methods: In a head-to-head comparison, the "self-organized" approach (learning from self-generated games) achieved performance gains comparable to GEPA, a human-designed optimization loop that uses real, in-distribution games.
  • Generational Gap: This "meta-capability" is a recent emergence. While newer models like Claude Opus-5 and GPT-6 Astra succeeded, older generations (like Claude Opus-4.6 or GPT-5.6-Sol) failed to show meaningful self-improvement in this autonomous setting.
Comparison showing that self-organized learning on self-generated games matches the performance of prescribed optimization on real games.

Limitations and Future Work

  • The Plateau Effect: The authors observed diminishing returns after approximately 20 context updates. It remains unclear if this is a limit of context-based learning or the models' current reasoning depth.
  • Causality: While the study correlates SRL strategies with success, it doesn't fully disentangle whether these strategies cause the improvement or are simply symptoms of a more capable underlying model.
  • Safety and Alignment: The ability of models to autonomously rewrite their own "operating manuals" and practice environments raises significant safety questions regarding misaligned self-improvement that the authors suggest need further scientific characterization.
  • Domain Specificity: While the results transfer across different game types, the study is centered on gameplay. Future work could explore if these autonomous loops emerge in more open-ended scientific or software engineering domains.

Summary updated

Abstract

LLM agents increasingly drive their own learning. Self-improvement methods delegate various components of model optimization to the models themselves. As models become more capable, this motivates a basic question: do these increasingly capable models still need us to specify how they should self-improve? In this paper we study an autonomous paradigm in which, given basic primitives, resources, and a general goal for self-improvement, agents construct their own self-learning algorithm for context-based optimization. Using the setting of gameplay as a testbed, a frozen LLM agent is given a compute budget and tools for collecting experience and updating text documents, and allowed to freely experiment without an externally imposed loop or access to real-world data. We find that sufficiently capable models do self-improve under these conditions, demonstrating strategies that align with classic cognitive frameworks for self-regulated learning and constructs from machine learning literature on self-generated curricula. Furthermore, we study the extent to which the learned text artifacts transfer to external benchmarks, observing improvement across different game horizons, mechanics, and 3D navigation.
BibTeX
@misc{sundaram2026do,
  author = {Shobhita Sundaram and Cansu Sancaktar and Phillip Isola},
  title = {{Do LLM Agents Still Need to Be Told How to Self-Improve?}},
  year = {2026},
  eprint = {2611.23219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23219},
}
Figure