A Transformer can be trained to act as a reinforcement learning agent that improves inside its own context window. This is called in-context reinforcement learning: the model's weights stay frozen at test time, it reads its own history of states, actions, rewards and safety costs, and it chooses better actions as that history grows. The most common recipe teaches this skill by imitation. One records many "learning histories", each a log of some ordinary learning algorithm going from bad to good on one task, and trains the Transformer to predict the next action in those logs. Ilya Zisman et al. showed that such histories can be faked cheaply: take a good policy and add random action noise that shrinks over time. The consequence matters for safety. The model inherits the style of improvement it imitated. If the imitated learner explored by injecting uniform noise in every state, including states next to a cliff, the in-context agent will do the same during its first test episodes. That is exactly when it knows least about the new task.
We propose to fix in-context safe exploration at its source, by changing the algorithm whose learning histories the Transformer imitates. The replacement is a learner built to explore safely while it learns. Each step is a small, size-controlled improvement measured in distribution space; this is natural policy gradient, or equivalently policy mirror descent, a step that moves the action distribution by a bounded KL divergence instead of a bounded parameter change. Its constraint enforcement is a primal-dual price on safety cost. It also follows a trajectory-level entropy budget that spends randomness in open regions and withholds it near hazards. The claim is that a frozen Transformer distilled from these histories makes fewer safety violations during test-time adaptation, not only after it, than one distilled from noise-schedule histories or trained with a safety-aware objective alone. A further claim is that its in-context behaviour measurably follows the step geometry of the learner it copied.
Core Claim
The closest existing system is Safe In-Context Reinforcement Learning (SCARED). It conditions the Transformer on the remaining cost budget and meta-trains with an exact-penalty dual objective, which avoids unstable Lagrange multipliers. SCARED changes the training objective and the conditioning signal. It says nothing about the exploration dynamics in the data, and the file notes that it still permits finite violations during test-time rollouts. Our proposal leaves the architecture and objective alone and changes the data-generating learner. The two are complementary, and SCARED's budget conditioning can sit on top of our histories.
The hypothesis is that an in-context learner's cumulative violations during adaptation track the cumulative violations of the source algorithm whose histories it distilled. If so, the right source is one with good cumulative behaviour, not just a good endpoint. Three ingredients make a source learner suitable, and each comes from a different direction in the file.
The first is a primal-dual natural policy gradient. Ding et al. prove that natural-gradient primal-dual updates converge on constrained problems at $O(1/\sqrt{T})$ for both reward gap and constraint violation. Its updates are multiplicative in probability space. An action that is observed to be costly loses probability geometrically, and nothing restores it except evidence. Each step moves the policy a controlled KL distance, which gives the Transformer a regular, learnable improvement increment.
The second is a trajectory entropy budget with separate reward and entropy critics. Zhan et al. show that budgeting entropy over a whole trajectory, instead of applying a uniform per-step bonus, makes policies nearly deterministic in precarious states and more random in stable ones.
The third is the observation that this combination has a closed form in small discrete problems, so clean histories can be produced exactly. For a tabular policy $\pi_k$ at round $k$, the entropy-regularized primal-dual mirror-descent step is
$$\pi_{k+1}(a\mid s)\ \propto\ \pi_k(a\mid s)^{\,1-\eta\alpha(s)}\exp\!\big(\eta\,[Q_r^{\pi_k}(s,a)-\lambda_k Q_c^{\pi_k}(s,a)]\big),$$
where $\eta$ is the step size, $Q_r$ and $Q_c$ are reward and cost action-values, $\lambda_k$ is the dual price on cost, and $\alpha(s)$ is a state-dependent entropy weight. The weights $\alpha(s)$ are set by a trajectory-level entropy budget computed from the separate entropy critic. The dual price follows a projected update $\lambda_{k+1}=\max(0,\lambda_k+\beta(\hat J_c-d))$, where $\hat J_c$ is the estimated cumulative cost and $d$ the budget. Optionally, a boundary-seeking residual term in the spirit of Fan et al. can be added so that histories end on the budget rather than deep inside it.
Research Program
We will build a history generator, the "safe mirror learner", for each benchmark family. In SafeDarkRoom it is the exact tabular update above, run with estimated Q-values from a modest number of rollouts per round. This gives realistic noise and keeps the source honest about what it observed. In the continuous SafeDarkMujoco and SafeVelocity tasks it is a Gaussian-policy natural-gradient primal-dual learner. Its natural step is computed with the randomized estimator of RAT, which avoids forming the Fisher matrix. Entropy is again allocated by a separate entropy critic under a trajectory budget.
From each generator we log thousands of learning histories across the training-task distributions defined by SCARED. These are tuples of state, action, reward and cost, with the remaining budget appended. We then train one causal Transformer per data source by next-action prediction, holding everything else fixed: architecture, token budget, optimizer and training steps. At test time the weights are frozen, the model is placed in unseen tasks, including SCARED's out-of-distribution edge-obstacle layouts, and it accumulates its own context.
The analysis has two layers. Behaviourally, we measure safety and return across the whole adaptation curve, not only at the end. Mechanistically, we borrow the diagnostic of Chernyavskiy et al., who showed that attention can implement entropy-regularized mirror descent and that a Transformer's belief path follows the Fisher-Rao geodesic with ratio about 1.1, against about 2.0 for a Euclidean mirror-descent baseline. We compute the KL distance between the in-context policy at successive episodes, the ratio of path length to geodesic length, and policy entropy as a function of distance to the nearest hazard. Their result is restricted to opponent modelling with a stripped-down Transformer. Ours tests whether the same geometry appears in closed-loop constrained control when it is put into the data rather than proved for a special architecture. If the in-context step sizes do not track the source $\eta$, the "inherited geometry" story fails even if safety improves, and we will report it that way.
One risk is the safety paradox described by Breaking Safety Paradox with Feasible Dual Policy Iteration: suppressing risky exploration starves the learner of evidence about where hazards are. An in-context agent that never touches a hazard may never learn its location from context. The trajectory entropy budget is our lever against this, since it permits deliberate, rationed probing. We will sweep the budget and measure how well the Transformer's implied cost estimate predicts true cost in out-of-distribution layouts.
Baselines
The naive reference point is AD$^\varepsilon$ (Zisman et al.) with cost tokens added to the context. Histories come from a decreasing uniform-noise schedule around per-task safe expert policies. This is the "do nothing about exploration style" choice, and it shows what data structure alone contributes. It uses the same experts' final policies as our generator, the same number of histories and tokens, and the same Transformer.
The strongest existing method is SCARED (Safe In-Context Reinforcement Learning), run with its exact-penalty objective and remaining-budget conditioning on its own benchmarks. It is the only method in the file built for frozen-weight safe adaptation, and it reportedly beats both earlier in-context agents and safe meta-learning baselines. We found no reported numerical result on Lacuna for SCARED on SafeDarkRoom, SafeDarkMujoco or SafeVelocity; the paper reports only qualitative claims and figures. We will therefore reimplement it and use the same data budget, training compute and hyperparameter search budget for every variant. Beating it means lower cumulative cost over the first test episodes at equal or higher final return across matched budget levels, with fewer budget-exceeding episodes. Because SCARED's objective is compatible with our data, we also run SCARED trained on our histories. If that combination is the best of all, the data contribution stands even though our standalone variant does not win outright.
As a non-in-context reference we run per-task PPO-Lagrangian and natural-gradient primal-dual learners directly on each test task. For PPO-Lagrangian, Safety Gymnasium reports a 98% reduction in safety violations relative to unconstrained PPO with a 45% drop in total reward on velocity tasks. This shows what a parametric safe learner costs in reward. It also bounds what the distilled agent could hope to match, since the source learner itself is the natural ceiling on safety during learning.
Three ablations isolate the key ingredient. In the first, histories come from the same primal-dual natural learner with uniform entropy, so state-dependent allocation is removed. In the second, histories come from first-order PPO-Lagrangian, which removes the KL-bounded multiplicative geometry while keeping constraint awareness. In the third, AD$^\varepsilon$ histories have every episode that exceeded budget rejected. This keeps "safe data" while removing structured exploration.
Why not just…?
Why not just filter unsafe episodes out of noise-schedule histories? This is the most serious threat, because it would deliver "safe data" with no new learner. Filtering does not change how the remaining histories explore: they still inject uniform noise in hazard-adjacent states and survived by luck. Filtering also discards precisely the episodes that carry hazard information, which makes the safety paradox worse. The rejection-filtered ablation is designed to settle this. If it matches the safe mirror learner on violations during adaptation in out-of-distribution layouts, we will conclude that the structure of exploration does not matter and drop that claim.
Why not just use SCARED's budget conditioning, which already makes the agent cautious under tight budgets? Budget conditioning tells the model how much risk it may take, but its learned exploration still comes from whatever histories it was trained on. SCARED reports adaptive caution, yet the file notes it still incurs finite violations during test rollouts. Training SCARED on both data sources separates the two effects directly.
Why not just distill PPO-Lagrangian histories, the standard safe learner? Safety Gymnasium documents oscillating multipliers that swing policies between reckless and timid behaviour. Fan et al. report that standard dual methods settle well inside the feasible region. Oscillating histories are both less safe while learning and harder to imitate as a consistent improvement rule. Still, this is an empirical question, and the PPO-Lagrangian-history ablation answers it. A tie would mean the natural-gradient component is unnecessary.
Why not just put an external safety shield on an ordinary in-context agent? Shields need a transition model. In the unseen, out-of-distribution layouts that motivate in-context adaptation, that model is wrong or missing, as discussed in Adaptive Probabilistic Shielding. Frequent interventions also stop the agent from internalizing why an action was unsafe (Reducing Safety Interventions). We include a shield given oracle hazard maps for in-distribution layouts only, as an upper reference.
Proposed Experiments
The main experiment runs every data source and baseline on SafeDarkRoom, SafeDarkMujoco and SafeVelocity. Each is tested at three cost budgets, in both in-distribution and SCARED's edge-obstacle out-of-distribution layouts, with at least five training seeds per variant. The primary metric is cumulative cost over the first several test episodes, the regime where in-context agents know least. Secondary metrics are final return, budget utilization measured as the residual between achieved cost and budget (following Fan et al.), and the fraction of episodes over budget. We also run larger Safety-Gymnasium navigation tasks (Point and Car Goal) as a transfer check. There, histories come from the continuous generator.
The decisive check holds the Transformer and objective fixed and changes only the history source. The safe mirror learner must reduce adaptation-time cumulative cost relative to both noise-schedule and rejection-filtered histories without lower final return, and the effect must survive in out-of-distribution layouts. The uniform-entropy ablation must show that state-dependent entropy accounts for part of the gain. Hazard-conditioned entropy curves, low near obstacles and higher in open space, must appear in the frozen model's behaviour rather than only in the source.
The mechanistic check requires the in-context per-episode KL steps to scale with the source step size $\eta$ when we train on histories generated at two different $\eta$ values. We also expect a geodesic ratio closer to 1 than for PPO-Lagrangian-distilled models. If safety improves but these geometric signatures are absent, we report a data-curation result rather than an inherited-algorithm result.
References
- Emergence of In-Context Reinforcement Learning from Noise Distillation
- Safe In-Context Reinforcement Learning
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
- Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation
- Attention as Natural Gradient: In-Context Mirror Descent for Opponent Modelling
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
- Reducing Safety Interventions in Provably Safe Reinforcement Learning