In traditional reinforcement learning (RL), an agent learns a new task by adjusting its neural network weights through gradient descent algorithms like PPO or SAC. This process is often slow, computationally expensive, and notoriously unstable.
However, a powerful alternative paradigm has emerged: In-Context Reinforcement Learning (ICRL). Instead of updating model weights to internalize a new policy, a frozen sequence model (such as a Transformer) can act as a complete reinforcement learning agent entirely within its forward pass. By treating its context window as a dynamic memory buffer, the model processes past states, actions, and rewards to refine its strategy on the fly.
To understand why this works, we must look at how the mechanics of sequence modeling map directly onto the fundamental components of reinforcement learning.
The Context Window as a Replay Buffer
In standard RL, a replay buffer stores past transitions so the optimizer can calculate gradient updates. In ICRL, the transformer's context window is the replay buffer.
When an agent interacts with an environment, its history of observations, actions, and rewards is appended directly to the prompt as a continuous sequence:
$$\text{Prompt} = (s_0, a_0, r_0, s_1, a_1, r_1, \dots, s_t)$$
As Kefan Song et al. demonstrate, when a model is provided with this iterative feedback loop, it naturally begins to exhibit core RL characteristics during inference—such as balancing the exploration of new strategies with the exploitation of known high-reward actions.
Because the model's weights are frozen, the "learning" that occurs is entirely non-parametric. Ethan Brooks et al. formalized this mechanism through In-Context Policy Iteration (ICPI). They show that by treating the context window as a dynamic trajectory buffer containing both successful and unsuccessful attempts, the model can execute an implicit policy improvement operator. It evaluates its own past performance within the prompt and selects superior actions for subsequent steps, effectively running a policy iteration loop without a single weight adjustment.
How the Forward Pass Implements Learning Algorithms
It is tempting to view this behavior as mere pattern matching or clever retrieval. However, theoretical and empirical evidence reveals that the forward pass of a transformer actually implements classic reinforcement learning algorithms under the hood.
Consider how a transformer computes attention. The attention matrix allows tokens late in the sequence to aggregate information from earlier tokens based on their relationship. Jiuqi Wang et al. provided a "white-box" explanation of this phenomenon, proving that transformers naturally learn to execute Temporal Difference (TD) methods within their forward pass. As the interaction history in the context window grows, the model uses the attention mechanism to iteratively refine its internal value estimates, causing its prediction error to drop exactly like a traditional TD learner.
[ Past Transitions: (s, a, r) ] ---> [ Attention Layers ] ---> [ Refined Value Estimates ]
(Context Window) (Forward Pass) (Next Action Selection)
This internal algorithmic execution extends to value-based control as well. Jinmei Liu et al. introduced a framework for scalable in-context Q-learning, demonstrating that a multi-head transformer can simultaneously predict values and policies. By utilizing advantage-weighted regression within the forward pass, the model can "stitch" together distinct, suboptimal experiences present in its context window to discover an optimal policy for an unseen task.
How the Policy Improvement Operator Emerges
A vanilla language model or transformer does not inherently know how to perform TD updates or Q-learning in-context. This capability must be cultivated through meta-training, where the model is exposed to a wide variety of learning experiences.
The core intuition behind training an in-context learner is to frame reinforcement learning as a sequence prediction problem. If you train a causal transformer on sequences of interactions that are explicitly ordered by increasing performance, the model learns to associate historical performance gains with future action selection.
This is the mechanism behind the $AD^\varepsilon$ algorithm developed by Ilya Zisman et al.. By injecting a decreasing schedule of noise into synthetic data, they generate artificial "learning histories." When the transformer learns to predict the next action in these sequences, it internalizes the underlying policy improvement operator itself.
To make this operator generalize to entirely novel environments, the model must be meta-trained on a massive, diverse task space. This requires substantial data infrastructure:
- Large-Scale Datasets: Datasets like XLand-100B, introduced by Alexander Nikulin et al., provide billions of transitions across tens of thousands of distinct tasks to teach transformers how to optimize policies purely in-context.
- Algorithmic Generalization: As Louis Kirsch demonstrates, when a transformer is meta-trained on a sufficiently diverse set of tasks, it undergoes a phase transition, moving away from task-specific memorization and discovering a general-purpose learning rule.
When scaled up, this meta-in-context learning allows agents to achieve remarkable feats. For example, the Adaptive Agent (AdA) architecture developed by Jakob Bauer et al. leverages these exact principles to adapt to completely novel 3D tasks within minutes—a timescale comparable to human learning—without requiring any test-time gradient updates.
Generalizing Beyond Numbers: Language as a Policy Space
While traditional RL relies strictly on scalar rewards, framing reinforcement learning as an in-context sequence modeling problem allows us to swap out numerical rewards for natural language feedback.
Traditional RL: [ State, Action ] ---> [ Scalar Reward (e.g., +1.0) ] ---> Gradient Update
In-Context RL: [ State, Action ] ---> [ Textual Critique / Hint ] ---> Context Expansion
This linguistic formulation opens up new ways to optimize agent behavior:
- Learning from Critiques: Yao Fu et al. showed that in complex negotiation environments, models can improve their strategies by incorporating qualitative critiques from an AI "critic" directly into the prompt as few-shot demonstrations for the next round.
- Prompted Policy Search: Instead of searching for neural network weights, the model can search for optimal strategies or prompts. Yifan Zhou et al. developed Prompted Policy Search (ProPS), which reformulates policy optimization as an iterative reasoning task. The LLM reviews a history of previous parameter sets and their corresponding rewards, using linguistic reasoning to propose better parameters in the next iteration.
- Skill Libraries: Rather than keeping every single transition in a massive context window, agents can abstract successful behaviors into high-level instructions. Kolby Nottingham et al. proposed Skill Set Optimization (SSO), where an agent dynamically extracts successful action sequences, converts them into natural language "skills," and retrieves them into the prompt when facing similar environmental states.
By shifting the burden of optimization from weight updates to context-driven sequence modeling, In-Context Reinforcement Learning enables frozen models to act as dynamic, highly adaptive agents that can learn, explore, and self-improve entirely in real-time.