Forget the Mood, Keep the Hazard: Tracking Drifting Risk Tolerance in Motion Planning

Robots that plan for the dangerous tail of what might happen, rather than the average case, already exist. They expose a single dial that sets how cautious they are. Today an engineer sets that dial once and leaves it. The people who share space with these robots do not stay fixed: a user of a feeding arm gets tired, a passenger gets used to a car's driving, a teleoperator grows bolder as the shift goes on. A natural fix is to borrow from preference-drift learning, where old feedback is down-weighted so the system follows what the person wants now. Done naively, this forgets more than it should. The same down-weighting that drops a stale preference also drops the memory of a rare near-collision three hundred trials ago, and rare events are exactly what tail-risk planning exists to remember.

This note proposes a planner that keeps the two kinds of memory apart. What the human wants, meaning how much risk they will accept, is a single number inferred from their pairwise comparisons of executed trajectories and allowed to drift, with older comparisons fading. What the world is like, meaning how often and how badly things go wrong, is learned from all data and fades only when the robot sees evidence that the environment has changed. The claim is that this split tracks drifting users about as well as forgetting everything, while avoiding the slow loss of safety that uniform forgetting causes once incidents become rare.

Core Claim

Risk-aware planners use conditional value-at-risk (CVaR). CVaR at level $\alpha$ is the expected cost over the worst $\alpha$ fraction of outcomes. Small $\alpha$ is cautious and $\alpha = 1$ is the plain average. Planners treat $\alpha$, or an equivalent failure-probability bound $\Delta$, as a fixed operator input. This holds for terrain navigation in STEP, for off-road driving with learned traction uncertainty in EVORA, and for the single policy in Risk Conditioned Neural Motion Planning, which accepts any $\Delta$ at run time without retraining. That last paper provides the machinery we reuse: one planner can be steered along the whole caution spectrum online. What none of these works provides is a way to decide where to steer it.

On the preference side, NS-DPO weights a comparison collected at time $t_i$ by $\gamma^{T-t_i-1}$ and recovers from preference flips in language-model alignment. It is risk-neutral, and its discount touches every piece of data equally. RA-PbRL learns CVaR-optimal policies from trajectory preferences with regret guarantees. It assumes a fixed true reward, so it cannot follow drift. Ra-DPO adds nested CVaR to token-level alignment on stationary offline data. Read together, these papers point to an unexamined design choice: which parts of a risk-sensitive preference model should be allowed to forget.

Our answer can be written in two lines. Let $p_\phi$ be the robot's learned model of cost outcomes along a trajectory $\tau$, including rare hazards. The planner executes

$$\tau_t = \arg\min_\tau \; \mathrm{CVaR}_{\hat\alpha_t}\big(C(\tau)\big),\quad C \sim p_\phi(\cdot \mid \tau),$$

where $p_\phi$ is fit on all experience and is discounted only when a change detector on hazard statistics fires. The tolerance estimate $\hat\alpha_t$ maximizes a discounted Bradley–Terry likelihood. A human with true tolerance $\alpha_t$ prefers trajectory $\tau^1$ over $\tau^2$ with probability

$$P(\tau^1 \succ \tau^2) = \sigma\!\Big(\beta\,\big[\mathrm{CVaR}_{\alpha_t}(C(\tau^2)) - \mathrm{CVaR}_{\alpha_t}(C(\tau^1))\big]\Big),$$

where $\sigma$ is the logistic function, $\beta$ is a rationality constant, and comparisons are weighted by $\gamma^{T-t_i-1}$ as in NS-DPO. The comparison is driven by time and effort as well as tail exposure, so the cost $C$ includes both. Because $\alpha$ is one number, it is identifiable from a few comparisons when the two trajectories differ in tail shape. We choose queries for this property, using pairs the risk-conditioned planner generates at two candidate $\alpha$ values.

The new element is not tracking $\alpha_t$ by itself, which a filter could do. It is the claim, tested below, that recency discounting belongs on the tolerance parameter and not on the hazard model. A system that discounts both forgets hazards as a predictable function of how long things have been quiet.

Research Program

We will build the method on top of a risk-conditioned soft actor-critic planner. The planner follows Huang et al., including its recursive critic for trajectory-level failure probability, with CVaR level as the conditioning input. The hazard model $p_\phi$ is an ensemble or evidential cost predictor in the spirit of EVORA, so that unfamiliar situations raise the estimated tail. The tolerance estimator is a discounted maximum-likelihood or particle filter over $\alpha$ fed by pairwise feedback.

The analysis has two parts. First, a tracking bound for $\hat\alpha_t$ under a total-variation budget, adapting the dynamic-regret argument of NS-DPO to a one-dimensional parameter. Second, a simple "hazard recall" statement. Suppose a hazard of probability $p$ was observed and the robot then sees $k$ quiet episodes. Under uniform discounting the effective weight of that observation decays as $\gamma^k$, so the estimated tail mass falls below $\alpha$ after roughly $\log(\alpha/p)/\log\gamma$ episodes, and the planner stops avoiding the hazard. Under change-triggered discounting, it does not.

The known weakness is the opposite failure. When the world really does change (the ice melts), an undiscounted hazard model stays too cautious until the change detector fires. We will measure this delay and not assume it away. Computing the Bradley–Terry term needs CVaR estimates of both compared trajectories under the current $p_\phi$. The planner's critic already supplies these, so no extra rollouts are needed.

Human drift is simulated with scripted users whose $\alpha_t$ follows three patterns: gradual fatigue (drifting more cautious), habituation (drifting bolder), and sudden flips. Their comparisons are generated with noise from the model above. We also include a deliberately misspecified user whose drift is in preferred speed rather than tail tolerance, to see when a single-$\alpha$ model breaks. Rare hazards are injected on a schedule with long quiet stretches. A small physical-style study comes last, conditional on simulation results.

Baselines

The naive reference is a fixed-$\alpha$ risk-conditioned planner tuned once at the start. It shows the cost of ignoring drift. A fixed conservative setting is included because it represents the "frozen robot" end of the trade-off. In CARLA unprotected left turns, a conservative worst-case planner needed 9.60 s against 8.64 s for the risk-bounded planner of Online Risk-Bounded Motion Planning, while a risky baseline collided 44% of the time.

The strongest competitor is uniform recency discounting applied to the whole model, meaning NS-DPO-style weights $\gamma^{T-t_i-1}$ on both the tolerance likelihood and the hazard model fit. It is the established method for preference drift: on the reward-flip benchmarks of NS-DPO, it keeps accuracy above 60–70% after a flip, where static DPO falls to about 50–55%. It is also the method most likely to track drift as fast as ours. We give it the same planner, data stream, query budget, and an equal hyperparameter search over $\gamma$. Beating it means matching its $\alpha$-tracking error within confidence intervals while showing a lower hazard-violation rate after long quiet periods.

A second strong competitor is stationary risk-aware preference learning in the style of RA-PbRL. It is CVaR-aware and never forgets, so it should be safe but slow to follow drift. On Assistive Gym we also compare against PbARL. PbARL reports 88% feeding success, against 41% for direct preference fine-tuning, with static synthetic personas. We give it the same comparison budget, retraining its preference model on a sliding window.

Two ablations isolate the key ingredient. The first discounts the hazard model by time while keeping our tolerance estimator; it should reproduce the forgetting failure. The second does not discount the tolerance likelihood; it should reproduce the slow-tracking failure. An oracle that reads the scripted $\alpha_t$ directly gives the upper bound.

Why not just…?

Why not just give the user a caution slider? This is a real threat. If users can state their tolerance directly and do so reliably, inference from comparisons adds little. The oracle-$\alpha$ condition is exactly that slider with a perfect user, so it bounds how much our method could gain. Two arguments against relying on the slider remain open questions. Fatigue drift is often unreported. The evidence from Liu et al. is that users refine their standards while interacting rather than holding them in advance; that study concerns binary feedback to an opaque learner, which is a different setting. In the physical study we will run a slider condition and report whether it wins. If it does, our contribution narrows to the hazard-memory split, which a slider-driven planner also needs.

Why not just use a sliding window or NS-DPO discounting on everything? This is our main baseline, and the hazard-recall experiment is designed to separate the two methods. The prediction is quantitative: uniform discounting starts violating hazards after roughly $\log(\alpha/p)/\log\gamma$ quiet episodes. If it does not, because a hazard's effect on the critic persists through other routes, our central claim fails.

Why not just detect change points and restart the preference model? Periodic restarts help even in stationary settings for collaborative filtering, as shown by COLLABORATIVE. A restart that also clears the hazard model creates a re-exploration phase that is dangerous in physical systems. A restart limited to $\alpha$ is in effect a version of our method with a different forgetting schedule. We will include it as a variant and report which schedule tracks sudden flips better.

Why not just widen safety margins permanently? Fixed conservatism avoids forgetting but pays in efficiency and usefulness. The CARLA gap of 9.60 s against 8.64 s quantifies this, and Assistive Gym reports that rigid assumptions about the human left dressing success at 26% against 89% with an active human model. The fixed-conservative baseline reports this cost in our own setting.

Proposed Experiments

Experiments run on three testbeds. The first is the OneObstacle, TwoRooms, and FlyTrapBig mazes from Huang et al., extended with hidden low-probability hazard patches. That paper reports execution risk staying within the requested bound; we found no reported numbers for drifting bounds. The second is Assistive Gym feeding, with scripted fatigue drift and rare sudden head motions that raise contact-force risk. The third is CARLA unprotected left turns, with drivers whose tolerance habituates across episodes.

The metrics are $\alpha$-tracking error, the simulated user's preference regret, task success and completion time, and the hazard-violation rate as a function of episodes since the last incident. This last curve is the decisive plot.

Three checks decide the outcome. First, against uniform discounting at its best $\gamma$, our violation curve must stay flat while the baseline's rises after quiet stretches, and tracking error must stay comparable. Second, after a scripted environment change, we report how many extra conservative episodes our change-triggered hazard memory costs compared with uniform discounting. This is the price of the design, stated openly. Third, for the misspecified speed-drifting user, we report whether a single $\alpha$ still explains comparisons better than chance. If it does not, the model needs a second preference dimension, and we will say so.

For reference points, PbARL's 88% feeding success and the 8.64 s / 100% CARLA left-turn result are the bars our method must not fall below while adapting. No reported result was found on Lacuna for any planner under drifting risk tolerance, so the drift-tracking numbers will be new.