Most safe reinforcement learning (RL) gives an agent a fixed safety budget for each episode: keep the expected sum of discounted "cost" (collisions, overheating, speed violations) below a number $\epsilon$. The standard solution trains a policy with a penalty weight, a Lagrange multiplier, that trades reward against cost. This note argues that such a policy has the same defect behavioral economists study in procrastinating humans. It is time-inconsistent: the plan it adopts at the start of an episode is no longer the plan it would choose halfway through. Suppose the episode is interrupted, re-planned, or judged from an intermediate state. The constraint then fails. Some trajectories have already spent most of the budget and others have spent almost none, yet the policy cannot tell which kind it is on. Carpentier et al. show this exactly in a 12-stage dam-management problem. Their Lagrangian policy targets a 0.90 terminal success probability, but restarted at stage 3 it reaches only 0.833. A state augmented with a running conditional expectation of the constraint restores 0.902. Their fix assumes known dynamics and exact dynamic programming. No model-free, exploring agent uses it.
We propose a deep RL agent that carries a promised budget: the amount of expected future cost it has committed to spend from its current state. At every step it learns how to divide the remaining budget among the possible next states. It can spend more risk in branches that turned out well and less in branches that turned out badly, while the average promise stays exact. This is the model-free version of Carpentier et al.'s martingale state. We use the time-inconsistency literature for its concepts: naive versus sophisticated planners, and the impossibility of consistent aggregation when objectives use different discount factors. Those concepts tell us what to measure and where standard methods break. In particular, the same mechanism handles the common but untreated case where safety must be judged more patiently than reward ($\gamma_C > \gamma_R$). Pitis proves that any fixed penalty-weighted sum is inconsistent in that case.
Core Claim
A constrained policy conditioned on a learned, mean-preserving promised budget will satisfy the safety constraint conditionally, from any intermediate state it reaches. Lagrangian and penalty-based methods do not. The new agent should also earn more reward than the simple fix of tracking "budget minus cost so far," whenever costs are stochastic. In deterministic-cost environments the two fixes coincide. We say this upfront because it bounds the claim.
Let $s_t$ be the state, $c_t$ the cost at step $t$, $\gamma_C$ the cost discount, and $z_t$ the promised budget, with $z_0=\epsilon$. Consistency requires the promise to be kept in expectation at every step:
$$z_t = c_t + \gamma_C\, \mathbb{E}\left[z_{t+1} \mid s_t, a_t, z_t\right].$$
The policy $\pi(a \mid s, z)$ and a reward critic under its own discount $\gamma_R$ are both conditioned on $z$. The next promise is produced by a learned allocator:
$$z_{t+1} = \frac{z_t - c_t}{\gamma_C} + \delta_\phi(s_t,a_t,s_{t+1},z_t) - m_\psi(s_t,a_t,z_t).$$
Here $\delta_\phi$ proposes how to shift budget toward or away from the realized next state. The term $m_\psi$ is a regression of $\delta_\phi$ onto $(s_t,a_t,z_t)$. Subtracting it makes the shift average to zero over next states, so the recursion above holds without a dynamics model. The allocator is trained to maximize the reward critic's value of the promised budget it hands to each next state. A learned minimum-achievable-cost critic sets a floor below which promises are clipped, so the agent cannot promise a budget it cannot meet.
Relation to Prior Work
Three papers motivate the gap. Carpentier et al. prove that expectation constraints make static multipliers time-inconsistent, and that a martingale budget state repairs this. They work with known stochastic transitions and do not learn or explore. Our contribution is to turn their martingale variable into a learned, sample-based allocator. Pitis proves that objectives with different discount factors cannot be combined consistently by a stationary Markovian scalar reward. He repairs this by tracking discount ratios in the state, but tests only an unconstrained toy problem. We carry his warning into constrained RL, where $\gamma_R \neq \gamma_C$ is natural but is usually ignored by setting the two discounts equal. Satija et al. see that trajectory constraints depend on history and use a backward value function to estimate expected cost-so-far. Their estimate is an average over histories, not the cost spent on the agent's own path. It yields a conservative per-state bound, and it assumes equal discounts. Our promised budget is specific to each path and is not an upper bound, which is where we expect to recover reward.
The skeptic's toolkit comes from the safe exploration literature. FOCOPS solves a penalty-balanced target distribution and projects the policy onto it. Ji et al. add PID control to the multiplier update to damp oscillation. Ding et al. prove global convergence for natural-gradient primal-dual methods. Each of these converges to a single multiplier for the start state, which is exactly the object Carpentier et al. show to be inconsistent. Feasible Dual Policy Iteration trains a deliberately reckless dual policy so the safe policy keeps seeing boundary data. It is the strongest training-time-safety method in the file, and it is orthogonal to our mechanism, so we treat it as both baseline and possible host.
The time-inconsistency literature supplies concepts but no method we can reuse. Its existing algorithms solve neighboring problems. Quasi-hyperbolic RL (Eshwar et al.) finds equilibria among temporal selves and notes that several equilibria may exist with very different payoffs. Pontryagin-guided policy optimization (Ko & Huh) avoids Bellman recursions under non-exponential discounting but needs a differentiable simulator. We use neither directly. Our agent is a "sophisticated" planner in the sense of Skalse et al.: it plans knowing its future self will re-evaluate. It reaches consistency through state augmentation rather than through a game between selves. Because each $(s,z)$ pair has one well-defined optimal action, this sidesteps the multiple-equilibria problem. We deliberately do not tackle hyperbolic discounting of reward. Two exponential discounts that differ already produce the inconsistency we target, and they keep Bellman recursions valid on the augmented state.
Research Program
We will build the allocator on top of a standard on-policy constrained learner (PPO with the SafePO implementations from Ji et al.) and an off-policy one (SAC). The goal is to show that the mechanism does not depend on the learner. Initial promises $z_0$ are sampled across a range of budgets during training. As a result, one policy serves every budget, much as TREBI conditions an offline diffusion planner on budget, but here online and without iterative denoising.
On the analysis side, we will prove a simple statement under exact critics and a correct mean-zero allocator. The constraint then holds conditionally from every reachable $(s,z)$. With equal discounts, the start-state value equals the CMDP optimum of the original problem, by reduction to Carpentier et al.'s extended dynamic program. We will also characterize what fails when $m_\psi$ is biased. A regression error of $\eta$ in the zero-mean correction gives a constraint error of at most about $\eta/(1-\gamma_C)$, since the error is added once per step and discounted geometrically. This links the size of the safety error to an error we can measure.
Baselines
The naive reference is PPO with a fixed reward penalty $r - \beta c$, tuned over a grid of $\beta$. Li et al. report that penalty shaping balances safety and progress worse than Lagrangian methods in MetaDrive. We include it to show where the conditional-consistency failure begins.
The standard competitor is PPO-Lagrangian with PID multiplier updates from Ji et al.. That paper reports a 98% reduction in safety violations relative to PPO, at a 45% drop in reward, on velocity tasks. We also run FOCOPS, which reports a Humanoid-Circle reward of 1106.1 within a cost limit of 50, against 671.0 for CPO and 697.5 for TRPO-Lagrangian. Beating FOCOPS here means matching its start-state return within confidence intervals while cutting conditional violations from intermediate states.
The strongest training-time-safety baseline is SAC-FDPI from Feasible Dual Policy Iteration. It reports near-zero violations across 14 Safety-Gymnasium tasks, but no per-environment numbers were found on Lacuna. We will rerun it from its description and treat its curves as the reference. The backward-value-function method of Satija et al. covers the "track average cost-so-far" idea; the file reports no tabular numbers for it.
The key ablation is a pathwise budget agent. It conditions on $\epsilon$ minus the discounted cost actually spent so far and has no allocator, so its promised budget is fixed by the realized path rather than learned. It isolates the one ingredient we claim matters, which is redistributing budget across branches.
All methods get the same environment steps, network sizes, and a matched hyperparameter search of equal size, each tuned on held-out seeds. We use ten seeds and report bootstrap confidence intervals. The proposal wins if it ties on start-state return and start-state cost while beating every baseline on restart consistency. It must also beat the pathwise ablation on return in environments with stochastic costs.
Why not just…?
Why not just condition the policy on budget remaining (ε minus cost so far)? This is the real threat. It fixes much of the inconsistency, and with deterministic costs it is the same as our method. The difference appears when costs are random. A pathwise budget treats an unlucky early cost as permanently spent and a lucky path as owed nothing. In effect it enforces something close to a per-path constraint, which is more conservative than the expectation constraint we were actually asked to satisfy. The dam problem settles the question exactly, since the extended dynamic program gives the true optimum. If the pathwise agent reaches roughly the 0.902 consistency and the same revenue as our allocator, we will narrow the claim to settings where branch-wise reallocation measurably pays.
Why not just keep the Lagrangian and add time and cost-so-far as input features? A single multiplier still prices cost the same way on every branch. Extra features let the policy react to history, but the training objective gives it no reason to keep a conditional promise. It optimizes only the start-state constraint. The restart-consistency test below measures exactly this. Carpentier et al.'s drop from 0.90 to 0.833 is the expected failure mode, and we will report whether added features close it.
Why not just use Pitis's discount-ratio state expansion for γ_R ≠ γ_C? It restores consistency for a fixed trade-off weight, but that weight still has to be tuned to hit a budget, and the budget is then met only from the start state. We include it as the consistent-scalarization baseline in the mismatched-discount experiments.
Why not just use FDPI, which already eliminates cost spikes? FDPI solves a different problem: the safe policy losing failure data late in training. It still converges to one start-state multiplier. We expect it to win on violations during training and lose on restart consistency, and we will also test an FDPI-plus-allocator combination.
Proposed Experiments
The first check is exact. We reimplement the 12-stage dam problem of Carpentier et al. and train every method model-free. Each policy is restarted at every intermediate stage, and we compare its conditional terminal-success rate with the published 0.833 (Lagrangian) and 0.902 (extended dynamic programming). Revenue is compared against the exact dynamic-programming optimum. If our learned agent gets within a few points of 0.902 from all stages, the model-free relaxation works.
The second check targets discount mismatch. We build a constrained version of Pitis's Procrastinator's Peril, where the reported naive plan valued at 3.20 collapses to a realized 1.00. We then run Safety-Gymnasium Point and Car navigation and MuJoCo velocity tasks with $\gamma_R=0.99$ and $\gamma_C=0.999$. In every case we measure start-state return and cost, and our main new metric, restart violation. To compute it, we sample intermediate states from each policy's own rollouts, compute the promise the method implies at that state, and record how far the realized future cost overshoots it. For Lagrangian methods the implied promise is their value estimate.
The third check concerns exploration-time safety. We count cumulative training violations and the fraction of violating samples in the replay buffer. FDPI reports this fraction at about 10x the baseline, which drops below 1%. This tests whether reallocating budget lets the agent probe the safety boundary on branches where it can afford to. The study is decisive if the allocator beats both the Lagrangian baselines and the pathwise ablation on restart violation, and beats the pathwise ablation on return under stochastic costs. If the pathwise ablation matches it, the allocator's added complexity is not justified and we will say so.
References
- Time Consistency for Multistage Stochastic Optimization Problems under Constraints in Expectation
- Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
- Constrained Markov Decision Processes via Backward Value Functions
- First Order Constrained Optimization in Policy Space
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Reinforcement Learning with Quasi-Hyperbolic Discounting
- Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
- Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting
- Safe Offline Reinforcement Learning with Real-Time Budget Constraints
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning