TL;DR
ACE-RL introduces a novel training paradigm called Experience-Handoff Reinforcement Learning (EHRL), where a single model is co-optimized to act, reflect on its failures, and curate "playbooks" of tips for its future self. By rewarding the model based on how its written insights actually improve subsequent attempts, ACE-RL achieves state-of-the-art performance on complex agentic benchmarks, proving that LLMs can learn to recursively improve their own task-solving strategies.
What Problem Does This Paper Address?
Training Large Language Models (LLMs) for agentic tasks (like coding or using APIs) typically relies on Reinforcement Learning (RL) from scalar rewards. While effective, this "outcome-only" approach is incredibly lossy: it compresses a rich trajectory filled with error logs, tool outputs, and discovered constraints into a single number.
Recent "Recursive Self-Improvement" (RSI) methods try to fix this by wrapping the model in a harness—a scaffold that lets the model reflect on a failed attempt and write notes (like a "playbook") to help the next attempt. However, these harnesses are usually fixed; the model is never actually trained to write better notes. It might reflect, but it doesn't learn how to reflect in a way that maximizes future success. The authors ask: Can a model learn, from task outcomes alone, both to act and to write the experience that its future self will inherit?
Methods
The authors propose ACE-RL (Agentic Context Engineering RL), which fixes a specific harness and trains one model to play three distinct roles:
- The Actor: Generates actions to solve the task.
- The Reflector: Analyzes a trajectory and extracts a key insight.
- The Curator: Edits a task-local "playbook" (a list of natural-language tips) based on the insight.
The core innovation is how these roles are rewarded. Instead of using a learned reward model for text, ACE-RL uses verifiable rewards. A curator's edit is rewarded by actually running a fresh actor rollout with the new playbook and seeing if the score improves.
To make this stable, they use a sampling tree (Figure 3) and a group-relative advantage estimation (similar to DeepSeek's GRPO). Crucially, they create comparison groups for every role:
- Reflections are compared against other reflections of the same trajectory.
- Edits are compared against other edits of the same insight. This isolates the "credit" for a good piece of text from the inherent difficulty of the underlying task.
Datasets
The method was evaluated across four diverse benchmarks:
- Mastermind: A logic-heavy code-breaking game.
- MBPP (Mostly Basic Python Problems): A standard Python coding benchmark.
- AppWorld: A complex environment requiring the agent to orchestrate multiple APIs (Spotify, Venmo, etc.) to solve day-to-day tasks.
- MiniSWEBench: A subset of the challenging SWE-bench, requiring agents to resolve real-world GitHub issues in a sandboxed repository.
Key Insights and Ideas
- Training for the Harness Matters: Simply putting a strong model (like a GRPO-trained actor) into a self-improvement harness isn't enough. ACE-RL shows that models specifically trained to use the harness and write for it significantly outperform those that aren't.
- Verifiable Intermediate Text: One of the biggest hurdles in RL for LLMs is rewarding intermediate reasoning or notes. ACE-RL bypasses the need for a "critic" or a reward model by treating the downstream task success as the ground truth for the quality of the "handoff" text.
- Generalization Beyond Training: During training, the model only sees up to 3 attempts (handoffs). However, at test time, the performance continues to climb even up to 8 attempts, suggesting the model learns a robust strategy for iterative refinement.
Key Results
ACE-RL consistently outperformed baselines across all benchmarks at equal training compute.
- AppWorld: ACE-RL achieved a success rate of ~0.67 (RSI@3), significantly higher than the base model (~0.25) and standard actor-only GRPO (~0.60 when put in the same harness).
- MBPP: It reached near-perfect performance (0.987), whereas standard RL methods plateaued much lower.
- Recursive Gain: As shown in Figure 2(a), ACE-RL's performance (orange line) starts higher and stays higher than actor-only methods as the number of attempts ($k$) increases.
Qualitatively, ACE-RL learned to write highly specific, actionable tips. For a task involving Venmo reminders, it learned to suggest: "+[1] 'Verify roommates' identities using supervisor or phone app data...'" after a failed first attempt, leading to success in the third attempt.
Limitations and Future Work
- Compute Overhead: While ACE-RL is efficient in terms of actor rollouts, the extra calls for reflection and curation add to the total generation cost during training.
- Task-Locality: Currently, playbooks are "task-local"—the model learns tips for a specific problem but doesn't yet transfer that "experience" to entirely different tasks.
- Scale: The experiments were conducted on models up to 4B parameters. Testing whether these "handoff" behaviors emerge or change at the scale of frontier models (70B+) remains an open question.
- Safety: Training models to autonomously improve their own context raises long-term safety questions regarding monitoring and control that the authors suggest need further study.
Summary updated