In-context reinforcement learning trains a sequence model, usually a Transformer, so that it improves its behavior on a new task just by reading its own past attempts in its prompt, with no weight updates at test time. The cheapest known way to teach this skill is to show the model fake "learning histories": noisy versions of a mediocre policy whose noise is gradually turned down, so that performance rises across episodes. This method, $AD^\varepsilon$, asks the model to imitate a curve of rising performance rather than a real learning algorithm. The trouble is that the fake histories begin with nearly random actions. In any task with hazards, random actions mean collisions, crashes and other safety violations. A model trained on these histories learns that early exploration is supposed to be dangerous.
We propose to make the noise itself respect a safety budget. When synthetic histories are generated in simulation, a random action is injected only if the cost already spent, plus the expected future cost of handing control back to the demonstrator, stays within a per-episode budget. Otherwise the demonstrator acts. Training histories then show reward rising while cost stays under budget, and histories generated under different budgets show more or less aggressive exploration. The model should learn a policy improvement procedure that explores only as far as its remaining budget allows. It would do so at roughly the data cost of $AD^\varepsilon$, not the cost of the online constrained meta-training that current safe in-context agents rely on.
Core Claim
The closest existing work comes from two directions, and each lacks what the other has. Safe In-Context Reinforcement Learning (SCARED) builds a budget-aware in-context agent. It conditions on cost feedback and remaining budget, and it meta-trains with an exact-penalty dual objective on SafeDarkRoom, SafeDarkMujoco and SafeVelocity. That training uses an online constrained-RL pipeline, which is exactly the compute the second line of work tries to avoid. Emergence of In-Context Reinforcement Learning from Noise Distillation shows that ordering trajectories from a single suboptimal demonstrator by decreasing $\varepsilon$-greedy noise is enough for a policy improvement operator to emerge. It reports a +120% gain over the best demonstrator policy in Dark Room. It has no notion of cost.
Our claim is about how the meta-training data is built, not about a new architecture. If noise injection is gated by a backward-plus-forward cost check, the resulting synthetic histories teach a frozen Transformer to explore safely. This is a stronger claim than simply labeling unsafe histories with their costs. The gated agent should match SCARED-style agents on return and constraint satisfaction during test-time adaptation, while using a small fraction of the meta-training environment steps and compute.
The gate borrows its logic from Constrained Markov Decision Processes via Backward Value Functions. That paper turns a whole-trajectory cost limit into a per-state check: cost-so-far plus expected future cost must not exceed the budget. It uses this check to keep a gradient-trained policy feasible throughout training. We use the same check in a different place: to decide which synthetic actions are allowed to appear in the dataset at all.
Research Program
Gate condition. Data generation follows $AD^\varepsilon$. For each training task we take a cheap, suboptimal but mostly safe demonstrator $\pi_d$, either scripted or briefly trained with a Lagrangian method, and roll it out under a noise level $\varepsilon_k$ that decays across episodes $k$. At step $t$ we draw a candidate random action $\tilde a$ with probability $\varepsilon_k$. We keep it only if
$$C_{<t} + c(s_t,\tilde a) + \hat V^c_{\pi_d}(s_{t+1}) \le b,$$
and otherwise we execute $\pi_d(s_t)$. The terms are defined as follows. $C_{<t}$ is the cost accumulated so far in the episode, which the simulator gives exactly, so the "backward" term needs no learned estimate. $c(s_t,\tilde a)$ is the immediate cost of the candidate action, and $s_{t+1}$ is the state it leads to. $\hat V^c_{\pi_d}$ is the demonstrator's expected future cost, estimated by short Monte Carlo rollouts in the simulator or by a small cost critic fit to demonstrator data. $b$ is the episode budget.
Budget conditioning. Budgets $b$ are sampled per history from a range. The model is conditioned on $b$ and on the running remaining budget, in the same way SCARED and Constrained Decision Transformer for Offline Safe Reinforcement Learning condition on cost-to-go. Because a larger $b$ admits more random actions, the dataset itself pairs budget with exploration aggressiveness.
Avoiding the safety paradox. Gating does not suppress all violations. Up to $b$, cost is allowed, so the data still contains transitions near hazards. This matters because of the "safety paradox" documented in Breaking Safety Paradox with Feasible Dual Policy Iteration: suppressing every violation starves the learner of the boundary data it needs to learn where danger is. We test this directly by including a zero-budget condition.
Model and accounting. The model is the standard causal Transformer used by In-context Reinforcement Learning with Algorithm Distillation, trained to predict the next action over multi-episode histories. We account for compute explicitly. For every method we log the environment steps and GPU-hours used to produce meta-training data, the training compute, and the per-step inference latency as a function of context length.
Secondary analysis. We also measure whether constraint behavior degrades late in long contexts. Technical Debt in In-Context Learning found an architecture-independent, roughly 1.5x efficiency penalty for in-context learning in many-shot passive regression. Nobody has checked whether the same penalty appears as budget overruns in long safe deployments.
Baselines
Do-nothing reference. The demonstrator $\pi_d$ run without any in-context learning gives the floor. A random policy gives the unsafe ceiling on cost.
Strongest existing method. SCARED, reproduced on the same three suites with its online exact-penalty meta-training, is the competitor that matters most. It is the only published in-context agent built for this exact question. The file gives no numeric results for it on any suite; only qualitative claims of outperforming ICRL and safe meta-RL baselines are available. We will therefore regenerate its numbers ourselves, using the same Transformer size, context length, task splits and evaluation budgets. The comparison is on two axes. Gated distillation "wins" if it matches SCARED's return and cumulative test-time cost within confidence intervals on out-of-distribution tasks while using at least an order of magnitude fewer meta-training environment steps. It also wins if, at equal compute, it beats SCARED on either return or cost without losing on the other.
Strongest offline safe sequence model. CDT is trained on the same gated data's demonstrator trajectories. For reference, it reports an average normalized reward of 82.99 and normalized cost of 0.72 on Bullet Safety Gym. That suite is not ours, so the figure only calibrates expectations. CDT tests whether multi-episode in-context improvement adds anything over single-episode budget conditioning.
Lagrangian reference. Parametric safe RL trained from scratch on each test task, PPO-Lag via SafePO from Safety Gymnasium, anchors sample efficiency. On velocity tasks that paper reports a 98% reduction in violations relative to PPO, with a 45% drop in reward.
Key ablation. The ablation that isolates our ingredient is ungated $AD^\varepsilon$ data relabeled with cost and remaining-budget tokens. It uses the same demonstrator, schedule, model and training steps. The only difference is whether unsafe random actions were allowed into the data.
Why not just…?
Why not just run $AD^\varepsilon$ and label each history with its costs, letting conditioning sort it out? This is the real threat. Return-to-go and cost-to-go relabeling worked well for CDT offline. If the ungated ablation, conditioned on a low budget, explores as safely as the gated model, then our central claim is wrong. Our reason to expect otherwise is distributional. Under a decaying-noise schedule, every ungated history begins with high cost. Low-budget prompts in early episodes would therefore ask the model about combinations it has never seen, the out-of-distribution conditioning failure the file attributes to cost-to-go tokens. The ablation decides this. We will report early-episode cost at matched final return, and if gating does not help, we will say so.
Why not just discard over-budget episodes from $AD^\varepsilon$ data? Filtering would leave mostly low-noise, late-curriculum episodes. That destroys the ordered improvement curve that $AD^\varepsilon$ identifies as the thing the model learns from, and it removes the boundary data the safety paradox says is needed. We include the filtered variant as a second ablation, at matched dataset size.
Why not just shield an unconstrained in-context agent at test time? A shield like Adaptive Probabilistic Shielding needs a model of the new task's dynamics, which is precisely what is unknown during in-context adaptation. Shield overrides also insert actions the model did not choose into its own context. We run an AD agent with an oracle simulator shield as an upper-bound reference. The shield has privileged information our agent lacks, so this comparison measures how much of the oracle's safety we recover without it.
Why not just pay for SCARED's online meta-training, since grid worlds are cheap? On SafeDarkRoom the compute gap may be small, which is why we report the steps-to-performance curve on SafeDarkMujoco and SafeVelocity, where it should widen. If the gap stays small there too, the computational motivation weakens. What would remain is a narrower but still useful result: a demonstrator-only route to safe in-context learning when online constrained RL is unavailable.
Proposed Experiments
Benchmarks and splits. We use SafeDarkRoom, SafeDarkMujoco and SafeVelocity, the suites introduced with SCARED, with its in-distribution and out-of-distribution splits. If code is unavailable we reimplement them from their published descriptions. We also run unconstrained Dark Room and Key-to-Door to confirm that gating does not destroy the +120% demonstrator improvement $AD^\varepsilon$ reports. Key-to-Door also probes the known subgoal-bottleneck failure: a hazard placed near the key tests whether gating makes that failure worse.
Metrics. We report return per test episode across in-context episodes 1 to $K$, and cumulative cost summed over all test episodes, which is the price paid while learning. We also report the fraction of episodes exceeding budget and the return-versus-budget curve as $b$ is swept at test time. The computational overhead is measured as meta-training environment steps, GPU-hours, and per-step inference latency against context length.
Decisive checks. The first decisive check is gated versus ungated-relabeled data at equal compute. Gated data should cut early-episode cost on out-of-distribution tasks, with no loss in final return. The second is gated distillation versus SCARED along a meta-training compute axis. The third is a budget sweep: return should rise monotonically with $b$, as SCARED's own Figure 2 shows for that method, while cost stays under $b$. Finally, a long-context run of many episodes tests whether budget overruns grow late in context, extending the technical-debt finding from regression to safe decision making.
References
- Safe In-Context Reinforcement Learning
- Emergence of In-Context Reinforcement Learning from Noise Distillation
- Constrained Markov Decision Processes via Backward Value Functions
- Constrained Decision Transformer for Offline Safe Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- In-context Reinforcement Learning with Algorithm Distillation
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning