Where Should LLM Agents Adapt at Test Time? In Context, in Weights, or Both

Parsa Mirtaheri, Mohammadreza Armandpour, Keivan Alizadeh, Iman Mirzadeh, Parshin Shojaee, Aryo Lotfi, David Harrison, Mehrdad FarajtabararXiv 2026 · arXiv

TL;DR

When LLM agents operate in persistent environments (like a specific codebase or database), they can learn from experience by either keeping past attempts in their context window (In-Context Learning) or updating their model weights via Reinforcement Learning. This paper demonstrates that while In-Context Learning (ICL) provides the most significant standalone performance boost, combining it with Reinforcement Learning (specifically GRPO) creates the most effective agents by balancing environmental efficiency with rigorous task verification.

What Problem Does This Paper Address?

LLM agents are increasingly deployed in long-term roles, such as a coding assistant for a specific GitHub repository or an analyst for a corporate database. These environments have a "latent structure"—specific coding idioms, database schemas, or API quirks—that an agent can learn over time.

However, there is a fundamental architectural question: How should an agent carry this experience forward? 1. In Context (ICL): Storing past interactions in the prompt. This is fast and requires no training but is limited by the context window. 2. In Weights: Updating the model's parameters (e.g., via fine-tuning or RL). This is permanent and doesn't take up prompt space but is computationally expensive and prone to "forgetting."

Until now, these two channels have mostly been studied in isolation. The authors seek to understand what each contributes, when they complement each other, and how they should be coordinated during "test-time adaptation"—where the agent learns solely from its own successes and failures without a human teacher.

Methods

The researchers compared 14 different adaptation mechanisms using a "test-then-train" protocol: the agent is evaluated on a new task before it is allowed to adapt using the results of that task.

  • In-Context Learning (ICL): The agent replays raw trajectories of past attempts. The authors tested different "retention" policies (FIFO vs. Prefix) and "admission" policies (only successful attempts vs. all attempts).
  • GRPO (Group Relative Policy Optimization): A reinforcement learning algorithm that updates a LoRA adapter based on verifiable rewards (e.g., did the code pass the unit tests?).
  • SDPO (Self-Distillation from Photo Optimization): A method where the model tries to match the token distribution of its own successful "sibling" attempts.
  • Hybrids: Combining ICL policies with weight-update rules (GRPO or SDPO).

The experiments were conducted using Qwen3.5-9B as the base model.

Datasets

The study evaluated agents across four distinct "worlds" (environments) totaling 109 tasks:

  • SWE-bench Verified (Xarray): Resolving real-world GitHub issues in a specific Python repository.
  • AppWorld (File System): Operating a simulated file system via API calls.
  • ScienceWorld (Measurement): Conducting experiments in a simulated laboratory.
  • BIRD-Critic-SQLite (Financial): Fixing faulty SQL queries in a financial database.

They also performed additional evaluations on scikit-learn tasks and a Toxicology database to ensure the findings generalized.

Key Insights and Ideas

  • ICL is the Heavy Lifter: If you can only choose one adaptation method, choose ICL. It provided a massive jump in success rates (from 28.8% to 49.4%) compared to weight updates alone.
  • The "Laziness" of ICL: Trace analysis revealed that ICL makes agents highly efficient—they perform far fewer documentation lookups and environment surveys because they "remember" the structure from the context. However, this efficiency comes at a cost: agents become overconfident and often skip crucial verification steps (like testing their code or querying the DB to check a result).
  • GRPO as a Behavioral Refiner: While GRPO alone isn't as strong as ICL, adding GRPO to an ICL-enabled agent yielded the best results (51.6%). Interestingly, GRPO doesn't just add knowledge; it "fixes" the laziness induced by ICL, encouraging the agent to perform task-specific verification while still benefiting from the environmental knowledge in the context.
  • SDPO's Mixed Bag: Unlike GRPO, self-distillation (SDPO) often reinforced the agent's existing shortcuts, leading to more "wrong" submissions rather than better verification.

Key Results

  • Standalone Performance: The stateless baseline started at 28.8% mean success. GRPO alone reached 37.0%, while the best ICL policy reached 49.4%.
  • The Winning Hybrid: The combination of ICL (Prefix-Successful) + GRPO achieved the highest overall mean success at 51.6%.
  • Efficiency Gains: ICL significantly shortened interaction lengths. In the AppWorld environment, ICL agents reduced API documentation lookups to nearly zero as they progressed through the task stream.
  • Verification Recovery: In the BIRD (SQL) benchmark, ICL agents dropped their "querying before submission" rate from ~90% to ~20%. Adding GRPO restored this querying behavior to ~68%, leading to fewer incorrect submissions.

Limitations and Future Work

  • Scale of Streams: The study looked at relatively short streams (dozens of tasks). It remains to be seen how weight updates and context management interact over thousands of tasks where "catastrophic forgetting" becomes a larger threat.
  • Model Specificity: The work focused on Qwen3.5-9B. Larger models or "reasoning" models (like the o1 series) might exhibit different adaptation dynamics.
  • Generalization vs. Specialization: The authors focused on online improvement within a specific environment. They did not measure whether these test-time updates degraded the model's general capabilities on unrelated tasks.
  • Future Direction: The authors suggest that future work should focus on "context management"—deciding exactly which experiences to keep in the prompt versus which to bake into the weights.

Summary updated

Abstract

LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.
BibTeX
@misc{mirtaheri2026where,
  author = {Parsa Mirtaheri and Mohammadreza Armandpour and Keivan Alizadeh and Iman Mirzadeh and Parshin Shojaee and Aryo Lotfi and David Harrison and Mehrdad Farajtabar},
  title = {{Where Should LLM Agents Adapt at Test Time? In Context, in Weights, or Both}},
  year = {2026},
  eprint = {2611.19700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19700},
}
Figure