TL;DR
When LLM agents operate in persistent environments (like a specific codebase or database), they can learn from experience by either keeping past attempts in their context window (In-Context Learning) or updating their model weights via Reinforcement Learning. This paper demonstrates that while In-Context Learning (ICL) provides the most significant standalone performance boost, combining it with Reinforcement Learning (specifically GRPO) creates the most effective agents by balancing environmental efficiency with rigorous task verification.
What Problem Does This Paper Address?
LLM agents are increasingly deployed in long-term roles, such as a coding assistant for a specific GitHub repository or an analyst for a corporate database. These environments have a "latent structure"—specific coding idioms, database schemas, or API quirks—that an agent can learn over time.
However, there is a fundamental architectural question: How should an agent carry this experience forward? 1. In Context (ICL): Storing past interactions in the prompt. This is fast and requires no training but is limited by the context window. 2. In Weights: Updating the model's parameters (e.g., via fine-tuning or RL). This is permanent and doesn't take up prompt space but is computationally expensive and prone to "forgetting."
Until now, these two channels have mostly been studied in isolation. The authors seek to understand what each contributes, when they complement each other, and how they should be coordinated during "test-time adaptation"—where the agent learns solely from its own successes and failures without a human teacher.
Methods
The researchers compared 14 different adaptation mechanisms using a "test-then-train" protocol: the agent is evaluated on a new task before it is allowed to adapt using the results of that task.
- In-Context Learning (ICL): The agent replays raw trajectories of past attempts. The authors tested different "retention" policies (FIFO vs. Prefix) and "admission" policies (only successful attempts vs. all attempts).
- GRPO (Group Relative Policy Optimization): A reinforcement learning algorithm that updates a LoRA adapter based on verifiable rewards (e.g., did the code pass the unit tests?).
- SDPO (Self-Distillation from Photo Optimization): A method where the model tries to match the token distribution of its own successful "sibling" attempts.
- Hybrids: Combining ICL policies with weight-update rules (GRPO or SDPO).
The experiments were conducted using Qwen3.5-9B as the base model.
Datasets
The study evaluated agents across four distinct "worlds" (environments) totaling 109 tasks:
- SWE-bench Verified (Xarray): Resolving real-world GitHub issues in a specific Python repository.
- AppWorld (File System): Operating a simulated file system via API calls.
- ScienceWorld (Measurement): Conducting experiments in a simulated laboratory.
- BIRD-Critic-SQLite (Financial): Fixing faulty SQL queries in a financial database.
They also performed additional evaluations on scikit-learn tasks and a Toxicology database to ensure the findings generalized.
Key Insights and Ideas
- ICL is the Heavy Lifter: If you can only choose one adaptation method, choose ICL. It provided a massive jump in success rates (from 28.8% to 49.4%) compared to weight updates alone.
- The "Laziness" of ICL: Trace analysis revealed that ICL makes agents highly efficient—they perform far fewer documentation lookups and environment surveys because they "remember" the structure from the context. However, this efficiency comes at a cost: agents become overconfident and often skip crucial verification steps (like testing their code or querying the DB to check a result).
- GRPO as a Behavioral Refiner: While GRPO alone isn't as strong as ICL, adding GRPO to an ICL-enabled agent yielded the best results (51.6%). Interestingly, GRPO doesn't just add knowledge; it "fixes" the laziness induced by ICL, encouraging the agent to perform task-specific verification while still benefiting from the environmental knowledge in the context.
- SDPO's Mixed Bag: Unlike GRPO, self-distillation (SDPO) often reinforced the agent's existing shortcuts, leading to more "wrong" submissions rather than better verification.
Key Results
- Standalone Performance: The stateless baseline started at 28.8% mean success. GRPO alone reached 37.0%, while the best ICL policy reached 49.4%.
- The Winning Hybrid: The combination of ICL (Prefix-Successful) + GRPO achieved the highest overall mean success at 51.6%.
- Efficiency Gains: ICL significantly shortened interaction lengths. In the AppWorld environment, ICL agents reduced API documentation lookups to nearly zero as they progressed through the task stream.
- Verification Recovery: In the BIRD (SQL) benchmark, ICL agents dropped their "querying before submission" rate from ~90% to ~20%. Adding GRPO restored this querying behavior to ~68%, leading to fewer incorrect submissions.
Limitations and Future Work
- Scale of Streams: The study looked at relatively short streams (dozens of tasks). It remains to be seen how weight updates and context management interact over thousands of tasks where "catastrophic forgetting" becomes a larger threat.
- Model Specificity: The work focused on Qwen3.5-9B. Larger models or "reasoning" models (like the o1 series) might exhibit different adaptation dynamics.
- Generalization vs. Specialization: The authors focused on online improvement within a specific environment. They did not measure whether these test-time updates degraded the model's general capabilities on unrelated tasks.
- Future Direction: The authors suggest that future work should focus on "context management"—deciding exactly which experiences to keep in the prompt versus which to bake into the weights.
Summary updated