Agents that must avoid rare disasters learn what a disaster looks like from very few examples. A self-driving policy might see a near-collision once in thousands of episodes. When the world changes, for example when pedestrians start moving faster or road friction drops, the usual fix is to throw away old experience and relearn. For an agent that cares about the average outcome this is wasteful. For an agent that cares about the worst outcomes it is close to fatal: the scarce failure data is discarded first, and the agent stays blind to new failure modes until it has crashed often enough to relearn them. This proposal argues that the change usually touches only one part of how the world works. If the agent models the world as separate cause-and-effect pieces (a causal model, in which each variable is produced by its own mechanism from a few parent variables), it can relearn only the piece that changed. It can then keep, and recombine through simulation, the rare-failure evidence held by every piece that did not change.
The objective throughout is Conditional Value-at-Risk (CVaR), the average return over the worst $\alpha$ fraction of episodes. Tail estimates are the most data-starved quantity in RL. That makes them the place where causal localization of change should pay off most, much more than for expected-return objectives, where prior causal non-stationary work has stayed so far.
Core Claim
After an abrupt change that alters a subset of mechanisms, a risk-averse agent with a factored causal world model can recover an accurate CVaR estimate and a safe policy faster than black-box alternatives. It does so by refitting only the changed mechanisms on post-change data, keeping the full history for the rest, and estimating the tail from simulated rollouts of the recombined model. The predicted advantage over global restarts, sliding windows, and context-conditioned critics should be largest when the post-change regime is a new combination of mechanism changes. It should be smallest when the agent re-enters a regime it has already seen. We state this prediction up front because it is also where the proposal could fail.
Concretely, with next-state variables $s'_j$, the model factorizes as
$$P(s' \mid s, a) = \prod_j P_j\big(s'_j \mid \mathrm{pa}_j(s,a)\big),$$
where $\mathrm{pa}_j$ are the causal parents of variable $j$ among current state and action. After a detected change affecting the set $C$ of mechanisms, each $P_j$ with $j \in C$ is refit on post-change data and every other $P_j$ keeps its full history. The policy then maximizes $\mathrm{CVaR}_\alpha(Z) = \mathbb{E}[Z \mid Z \le q_\alpha(Z)]$, where $Z$ is episode return and $q_\alpha$ its $\alpha$-quantile, estimated by rollouts in the recombined model. A direct CVaR estimate from $n$ real episodes rests on only about $\alpha n$ tail samples, so its error shrinks roughly like $1/\sqrt{\alpha n}$. The simulated estimate is limited instead by the error of the refit mechanisms. That is the source of the expected speedup.
Relation to Prior Work
The closest work is Piecewise-Stationary Combinatorial Semi-Bandit with Causally Related Rewards. It detects changes with likelihood-ratio tests and uses "group restarts" that reset estimates only for causally linked arms in a linear structural equation model, and it reports that this beats global resets. We take the localized-restart idea from it. We move it from a risk-neutral bandit with linear equations to a full Markov decision process with nonlinear mechanisms, and we argue it matters more for tail objectives than for expected regret.
Non-Stationary Structural Causal Bandits motivates the work. It shows that ignoring causal propagation across time is catastrophic, with regret around 1.9K against over 34K for a myopic baseline. However, it assumes the time-unrolled graph is known, its cost grows exponentially in variables and horizon, and it optimizes expectations. We learn structure and avoid unrolling.
Two lines supply method. Castor jointly learns hidden regimes and regime-specific graphs by expectation-maximization, reaching about 99% F1 on 40-variable synthetic data where CD-NOD stays below 12%. It works offline and without decisions, and we use it to initialize the mechanism library from logged data. The work of Hwang et al. (Quantized Local Independence Discovery, Fine-Grained Causal Dynamics Learning with Quantization) learns event-specific sparse dynamics that improve robustness. Its dynamics are stationary in time and its objective is the average return, which we replace with tail evaluation.
On the risk side, Predictive CVaR Q-learning gives step-wise Bellman updates for CVaR in stationary MDPs. WCSAC constrains a Gaussian distributional cost critic, and the quantile-regression motion planner maximizes a low return quantile in driving. None of the three models change.
Risk-averse Learning with Non-Stationary Distributions handles drift under CVaR with periodic global restarts and zeroth-order queries. It needs many samples per step, and that is the cost our localization is meant to avoid. It constrains our design: under slow, diffuse drift with no localized change, its black-box guarantees apply and ours may not.
Research Program
The system has three parts.
The first is a factored world model with one small network per next-state variable, each over a learned sparse parent set. It is initialized from offline data with a CASTOR-style regime and structure fit, then updated online.
The second is a change localizer. It runs a separate likelihood-ratio test on each mechanism's one-step prediction residuals, so a detected change names the mechanisms in $C$ rather than only signalling "something changed."
The third is a risk-averse planner. It optimizes CVaR by actor-critic training on rollouts from the recombined model, and uses quantile critics so the tail is not forced to be Gaussian.
Between a change and its localization there is a window in which the agent is unsure which mechanisms changed. During this window the planner takes CVaR jointly over episode randomness and over the plausible change sets, weighted by the localizer's test statistics. It becomes cautious in proportion to its uncertainty about structure, not uniformly.
On the analysis side, we will measure how tail recovery time scales with the number of changed mechanisms and with tail rarity $\alpha$. We will test the prediction that the causal method's recovery depends mainly on the former, and black-box methods' recovery mainly on the latter.
Baselines
The naive reference is a quantile-regression soft actor-critic in the style of the CQR-SAC planner, trained with no change handling. It shows how badly a stationary risk-averse agent degrades. Its stationary curved-road collision rate in SUMO is 3.71%, against 37.56% for plain SAC, as reported in that paper. This is the strongest reported tail-safety number in the file, and our method must at least match it before any shift.
The second baseline adds a global likelihood-ratio change detector and fully restarts critic and policy, as in the restart logic of Risk-averse Learning with Non-Stationary Distributions. A sliding-window variant of the same agent is the third baseline.
The strongest black-box competitor is a context-conditioned distributional critic. A recurrent encoder infers a latent regime from recent transitions and feeds it to quantile critics and the policy. It is trained across the same randomized regimes our model sees offline.
The oracle receives true regime labels and a regime-specific critic. It bounds achievable performance.
Two ablations isolate the key ingredient. The first uses the same model-based CVaR planner but refits the whole world model on change instead of only the changed mechanisms. This tests whether localized reuse is what helps. The second keeps localized reuse but plans for expected return. This tests whether the benefit is specific to tail objectives, as claimed.
Every method gets the same offline data, environment steps, network sizes, and hyperparameter search budget, with results over at least ten seeds. To count as beating a baseline, our method must show at least a 30% reduction in post-shift tail recovery time, or in transition-window violations, with non-overlapping confidence intervals, and without lower mean return than the context-conditioned critic.
No reported result exists on Lacuna for any method on non-stationary variants of these benchmarks. All post-shift numbers will be new.
Why not just…?
Why not just train a context-conditioned distributional critic across randomized regimes? This is the real threat. If every test regime appeared during training, a recurrent critic can recognize it and needs no causal model. Our claim rests on recombined shifts, meaning combinations of mechanism changes never seen together, where a black-box encoder has no stored tail for the combination but a factored model can simulate one. The experiments split test regimes into revisited and recombined. If the context critic matches us on recombined shifts, the central claim is false and we will report it as such.
Why not just restart globally, or use a sliding window, when a change is detected? Group restarts already beat global restarts for expected regret in Piecewise-Stationary Combinatorial Semi-Bandit with Causally Related Rewards. Risk-averse Learning with Non-Stationary Distributions shows that black-box CVaR learning after restarts needs many samples per step. Tail statistics should suffer most from discarded data because they rest on about $\alpha n$ samples. The recovery-time-versus-$\alpha$ curves test this directly. If global restarts recover as fast at $\alpha = 0.01$, the localization argument fails.
Why not just train one very conservative policy that is safe in all regimes? Uniform worst-case caution produces the frozen-robot behaviour noted in the motion planning paper. We include a domain-randomized policy maximizing CVaR over the union of regimes and compare mean return at matched tail safety. Our method should equal its violation rate while earning more in benign regimes.
Proposed Experiments
The first testbed is the stochastic grid-world and decision tree of Predictive CVaR Q-learning. PCVaR-Q reaches CVaR 2.45 against an optimum of 2.50 and 1.96 for a risk-neutral agent there. We make it piecewise-stationary by switching individual factored mechanisms, such as slip probability in one corridor or hazard penalty magnitude. With exact oracle CVaR available, we can measure estimation error over time.
The second testbed is Safety Gym's DynamicEnv from WCSAC. Hazard radius, robot friction, and goal placement change independently mid-training. The third is SUMO's occluded crosswalk and curved road, where pedestrian speed, emergence rate, and braking dynamics shift separately and in new combinations.
Structure recovery is checked on CASTOR's synthetic regimes, aiming to stay near its reported F1, so that downstream failures can be traced to planning rather than discovery.
The metrics are realized CVaR at $\alpha \in \{0.01, 0.05, 0.1\}$ per regime; tail recovery time, defined as episodes until realized CVaR is within 5% of the oracle; safety violations in the window between change and localization; mean return; and the localizer's precision at naming changed mechanisms.
Two checks decide the outcome. The first is whether the gap over the context-conditioned critic appears on recombined but not revisited shifts. The second is whether the gap over the whole-model-refit ablation grows as $\alpha$ shrinks, as the data-scarcity argument predicts. A third diagnostic injects diffuse drift that touches every mechanism slightly. Here we expect the advantage to vanish, which would mark the method's boundary honestly.
References
- Piecewise-Stationary Combinatorial Semi-Bandit with Causally Related Rewards
- Non-Stationary Structural Causal Bandits
- Castor: Causal Temporal Regime Structure Learning
- Quantized Local Independence Discovery for Fine-Grained Causal Dynamics Learning in Reinforcement Learning
- Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning
- Predictive CVaR Q-learning
- WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning
- Motion Planning for Autonomous Vehicles in the Presence of Uncertainty Using Reinforcement Learning
- Risk-averse Learning with Non-Stationary Distributions