Towards Monotonic Improvement in In-Context Reinforcement Learning

Wenhao Zhang, Shao Zhang, Xihuai Wang, Yang Li, Ying WenSubmission to ICLR 2026 · Rejected

TL;DR

In-Context Reinforcement Learning (ICRL) models like Algorithm Distillation often suffer from test-time performance regression due to "Contextual Ambiguity," where a few stochastic bad actions fool the model into believing it has reverted to a weaker policy. The authors introduce Context Value Informed ICRL (CV-ICRL), which conditions policy actions on an explicit estimate of the optimal achievable return for the current context. This breaks the vicious cycle of degradation, tightens theoretical performance bounds, and significantly stabilizes in-context policy improvement.

What Problem Does This Paper Address?

In ICRL, sequence models (typically causal Transformers) are trained on histories of learning trajectories generated by standard RL algorithms (such as PPO checkpoints exhibiting monotonic improvement). The goal is to produce agents that "learn how to learn" in-context without parameter updates.

However, during test time, these models frequently experience severe performance drops instead of improving monotonically. The authors diagnose this as Contextual Ambiguity:

  • Causal sequence models exhibit strong recency bias, relying heavily on immediate recent steps to infer which policy stage to emulate.
  • Test-time action sampling introduces randomness. A few unlucky, low-reward steps can make the recent context resemble early, sub-optimal training checkpoints.
  • Misidentifying its current competency level, the policy switches to a worse policy, generating even poorer data and triggering a vicious cycle of performance collapse.

Methods

To counteract ambiguous context histories, the authors introduce Context Value ($V_C$), defined as the theoretical maximum expected return achievable given context $C$ ($V_C = J(\pi^*_C)$). Because expanding contexts provide weakly more information, true Context Value is non-decreasing.

Conditioning the policy on $V_C$ gives it an explicit, unambiguous signal of performance quality:

  • Theoretical Guarantee: The authors prove that conditioning on $V_C$ strictly tightens the upper bound on the performance gap between the learned policy and the context-optimal policy compared to standard ICRL.
  • Practical Implementations (CV-ICRL):
  • Training Time: The true expected return of the checkpoint policy generating the trajectory, $J(\pi_i)$, acts as the target Context Value $\widehat{V}_C$.
  • Test-Time Option 1 ($\text{CV-ICRL-}\phi(C)$): An auxiliary prediction head is added to the Transformer to estimate $V_C$ directly from the sequence context via MSE loss. This adapts dynamically to task difficulty, though it can still inherit some context noise.
  • Test-Time Option 2 ($\text{CV-ICRL-}\phi(t)$): Sets $V_C$ via a deterministic, non-decreasing function of the elapsed timestep $t$ (e.g., $\min(t/1000, 0.95)$). This guarantees monotonic signaling and eliminates recency bias entirely, though it is less adaptive across tasks of varying difficulty.

Datasets

  • Dark Room: A standard sparse-reward 2D navigation gridworld used in prior ICRL benchmarks. Evaluated on 20 unseen goal locations.
  • Minigrid: A richer symbolic 2D gridworld suite with $7 \times 7 \times 3$ observations.
  • Standard Multi-task evaluation: Trained on 400 procedurally generated layouts and tested on 20 unseen seeds across 6 task types: LavaCrossingS9N3, LavaCrossingS9N2, SimpleCrossingS9N3, SimpleCrossingS11N5, BlockedUnlockPickup, and Unlock.
  • Out-of-Distribution (OOD) Task Generalization: Trained on the 6 standard task families and tested on 4 held-out task families: LavaCrossingS11N5, FourRooms, DoorKey-6x6, and DoorKey-8x8.
  • All training trajectories were generated by collecting rollout data across the training progression of PPO policies.

Key Insights and Ideas

  • Recency bias causes policy collapse: Attention heatmap visualizations confirm that Transformer agents attend disproportionately to immediate recent tokens. In pure sequence modeling, recent sub-optimal samples look identical to early training stages, causing the agent to "forget" its skill level.
  • Separating "What happened" from "How good I should be": Supplying an explicit value token prevents the agent from falling into the trap of purely pattern-matching historical reward quality.
  • First cross-task generalization in Minigrid: Prior work mostly validated ICRL on simple variations of reward positions (e.g., Dark Room). This work demonstrates that AD-like agents can genuinely adapt zero-shot across distinct structural game mechanics (e.g., generalizing from crossing lava to picking up keys and opening doors).

Key Results

The authors evaluated policies using Average Episode Return (AER), Last Episode Return (LER), and Degradation Frequency ($D_F$, the percentage of episodes where return dropped by $\ge 5\%$ compared to the previous episode):

  • Significant reduction in policy degradation: On BlockedUnlockPickup, baseline Algorithm Distillation had a degradation frequency of $2.49\%$ (with AD-$\epsilon$ collapsing at $83.98\%$ and IDT failing entirely), whereas $\text{CV-ICRL-}\phi(t)$ reduced it to $1.05\%$ while achieving a higher AER of $0.952$ (vs. $0.911$ for AD). On Unlock, degradation frequency dropped from $4.89\%$ (AD) down to $0.22\%$ ($\text{CV-ICRL-}\phi(C)$).
  • Cross-task-type Generalization: On unseen Minigrid tasks (e.g., DoorKey-6x6), $\text{CV-ICRL-}\phi(C)$ reached an AER of $0.745$ (compared to $0.464$ for AD) and cut degradation frequency roughly in half ($27.0\%$ vs $52.7\%$).
  • Ablations: Removing $V_C$ conditioning while keeping the auxiliary prediction loss collapses performance back to baseline levels, verifying that explicit value conditioning—not auxiliary multi-task representation learning—drives the performance gains.

Limitations and Future Work

  • Heuristic Test-Time Values: The deterministic schedule $\phi(t)$ requires manual tuning of hyperparameters (e.g., scaling slope and saturation point) depending on task horizon and difficulty.
  • Error Propagation in $\phi(C)$: When using the learned estimator $\phi(C)$, severe out-of-distribution interaction trajectories can still lead to noisy value estimates.
  • Future Directions: Developing dedicated uncertainty/error estimators for Context Value to produce confidence-weighted policy adaptations, and exploring Context Value conditioning for prompt-based in-context reasoning in LLM agents.

Summary updated

Abstract

In-Context Reinforcement Learning (ICRL) has emerged as a promising paradigm for developing agents that can rapidly adapt to new tasks by leveraging past experiences as context, without updating their parameters. Recent approaches train large sequence models on monotonic policy improvement data from online RL, aiming to achieve continued improvement in testing time performance. However, our experimental analysis reveals a critical flaw: these models cannot demonstrate continued improvement like the training data during testing time. Theoretically, we identify this phenomenon as *Contextual Ambiguity*, where the model's own stochastic actions can generate an interaction history that misleadingly resembles that of a sub-optimal policy from the training data, initiating a vicious cycle of poor action selection. To resolve the Contextual Ambiguity, we introduce *Context Value* into training phase and propose **Context Value Informed ICRL** (CV-ICRL). CV-ICRL uses Context Value as an explicit signal representing the ideal performance theoretically achievable by a policy given the current context. As the context expands, Context Value could include more task-relevant information, and therefore the ideal performance should be non-decreasing. We prove that the Context Value tightens the lower bound on the performance gap relative to an ideal, monotonically improving policy. We further propose two methods for estimating Context Value at both training and testing time. Experiments conducted on the Dark Room and Minigrid testbeds demonstrate that CV-ICRL effectively mitigates performance degradation and improves overall ICRL abilities across various tasks and environments. The source code and data of this paper are available at https://anonymous.4open.science/r/towards_monotonic_improvement-E72F.
BibTeX
@misc{zhang2025towards,
  author = {Wenhao Zhang and Shao Zhang and Xihuai Wang and Yang Li and Ying Wen},
  title = {{Towards Monotonic Improvement in In-Context Reinforcement Learning}},
  year = {2025},
  eprint = {2509.23209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.23209},
}
Figure