Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry YangarXiv 2026 · arXiv

TL;DR

ACE-RL introduces a novel training paradigm called Experience-Handoff Reinforcement Learning (EHRL), where a single model is co-optimized to act, reflect on its failures, and curate "playbooks" of tips for its future self. By rewarding the model based on how its written insights actually improve subsequent attempts, ACE-RL achieves state-of-the-art performance on complex agentic benchmarks, proving that LLMs can learn to recursively improve their own task-solving strategies.

What Problem Does This Paper Address?

Training Large Language Models (LLMs) for agentic tasks (like coding or using APIs) typically relies on Reinforcement Learning (RL) from scalar rewards. While effective, this "outcome-only" approach is incredibly lossy: it compresses a rich trajectory filled with error logs, tool outputs, and discovered constraints into a single number.

Recent "Recursive Self-Improvement" (RSI) methods try to fix this by wrapping the model in a harness—a scaffold that lets the model reflect on a failed attempt and write notes (like a "playbook") to help the next attempt. However, these harnesses are usually fixed; the model is never actually trained to write better notes. It might reflect, but it doesn't learn how to reflect in a way that maximizes future success. The authors ask: Can a model learn, from task outcomes alone, both to act and to write the experience that its future self will inherit?

Figure 1: Experience-Handoff RL. (a) Outcome-only RL compresses each trial into a scalar reward. (b) A harness runs several calls to the same model (cis; in ACE, actor, reflector, and curator) and preserves experience as a model-written context, such as a playbook, handed to the next attempt (blue), but the model never learns to write better playbooks. (c) Experience-handoff RL trains the model for the harness it runs in, from task outcomes alone. For example, ACE-RL rewards each handoff artifact for what it does to a future rollout: (i) each new context is scored by a fresh rollout, and (ii) each role is compared within its own group (stacked boxes), which lowers the variance of its credit.

Methods

The authors propose ACE-RL (Agentic Context Engineering RL), which fixes a specific harness and trains one model to play three distinct roles:

  1. The Actor: Generates actions to solve the task.
  2. The Reflector: Analyzes a trajectory and extracts a key insight.
  3. The Curator: Edits a task-local "playbook" (a list of natural-language tips) based on the insight.

The core innovation is how these roles are rewarded. Instead of using a learned reward model for text, ACE-RL uses verifiable rewards. A curator's edit is rewarded by actually running a fresh actor rollout with the new playbook and seeing if the score improves.

To make this stable, they use a sampling tree (Figure 3) and a group-relative advantage estimation (similar to DeepSeek's GRPO). Crucially, they create comparison groups for every role:

  • Reflections are compared against other reflections of the same trajectory.
  • Edits are compared against other edits of the same insight. This isolates the "credit" for a good piece of text from the inherent difficulty of the underlying task.

Datasets

The method was evaluated across four diverse benchmarks:

  • Mastermind: A logic-heavy code-breaking game.
  • MBPP (Mostly Basic Python Problems): A standard Python coding benchmark.
  • AppWorld: A complex environment requiring the agent to orchestrate multiple APIs (Spotify, Venmo, etc.) to solve day-to-day tasks.
  • MiniSWEBench: A subset of the challenging SWE-bench, requiring agents to resolve real-world GitHub issues in a sandboxed repository.

Key Insights and Ideas

  • Training for the Harness Matters: Simply putting a strong model (like a GRPO-trained actor) into a self-improvement harness isn't enough. ACE-RL shows that models specifically trained to use the harness and write for it significantly outperform those that aren't.
  • Verifiable Intermediate Text: One of the biggest hurdles in RL for LLMs is rewarding intermediate reasoning or notes. ACE-RL bypasses the need for a "critic" or a reward model by treating the downstream task success as the ground truth for the quality of the "handoff" text.
  • Generalization Beyond Training: During training, the model only sees up to 3 attempts (handoffs). However, at test time, the performance continues to climb even up to 8 attempts, suggesting the model learns a robust strategy for iterative refinement.

Key Results

ACE-RL consistently outperformed baselines across all benchmarks at equal training compute.

  • AppWorld: ACE-RL achieved a success rate of ~0.67 (RSI@3), significantly higher than the base model (~0.25) and standard actor-only GRPO (~0.60 when put in the same harness).
  • MBPP: It reached near-perfect performance (0.987), whereas standard RL methods plateaued much lower.
  • Recursive Gain: As shown in Figure 2(a), ACE-RL's performance (orange line) starts higher and stays higher than actor-only methods as the number of attempts ($k$) increases.
Performance curves showing ACE-RL's superior recursive improvement over 8 attempts on AppWorld.

Qualitatively, ACE-RL learned to write highly specific, actionable tips. For a task involving Venmo reminders, it learned to suggest: "+[1] 'Verify roommates' identities using supervisor or phone app data...'" after a failed first attempt, leading to success in the third attempt.

Limitations and Future Work

  • Compute Overhead: While ACE-RL is efficient in terms of actor rollouts, the extra calls for reflection and curation add to the total generation cost during training.
  • Task-Locality: Currently, playbooks are "task-local"—the model learns tips for a specific problem but doesn't yet transfer that "experience" to entirely different tasks.
  • Scale: The experiments were conducted on models up to 4B parameters. Testing whether these "handoff" behaviors emerge or change at the scale of frontier models (70B+) remains an open question.
  • Safety: Training models to autonomously improve their own context raises long-term safety questions regarding monitoring and control that the authors suggest need further study.

Summary updated

Abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
BibTeX
@misc{kim2026experience,
  author = {Doyoung Kim and Byeongguk Jeon and Kailin Zhang and Sherry Yang},
  title = {{Experience-Handoff Reinforcement Learning for Recursive Self-Improvement}},
  year = {2026},
  eprint = {2611.16409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16409},
}
Figure