Barriers on Calibrated Mode Sets for Safe Learning Around Unpredictable Agents

A robot learning by trial and error next to people or cars has to guess where those agents will go. Their futures branch. A pedestrian at a fork goes left or right, never down the middle. Forecasters that output a mixture of possible futures capture this branching. The standard way to turn such a forecast into a safety rule is a chance constraint: stay safe with probability at least $1-\delta$ under the predicted mixture. That rule depends on the forecaster's mixture weights, meaning its estimate of how likely each future is. When the forecaster was trained on little data, which is the normal situation in robotics, those weights are the part of the forecast most likely to be wrong. A confident 95/5 split between "yields" and "cuts across" can hide a true 60/40 split. A shield built on the 5% will under-protect exactly when the rare branch is taken.

We propose to build the safety shield on a set of plausible modes rather than on mode probabilities. Each mode that has not yet been ruled out gets its own control barrier function, a scalar "distance to danger" whose rate of decrease the controller is not allowed to exceed. The safety margin around each mode is calibrated online from how far the agent actually strays from that mode's predicted path. The shield loosens as the robot watches the agent and rules modes out. Because this evidence comes from observing the other agent, not from the robot approaching failure, the shield can relax during learning without the robot ever visiting the unsafe boundary. This escapes, for agent-induced uncertainty, the "safety paradox" in which a safe learner starves itself of the data needed to learn where danger is.

Core Claim

When the uncertainty that matters for safety is about another agent's intent, and the forecaster of that intent is trained on scarce data, a barrier defined over a calibrated set of surviving modes gives two things at once. It keeps the in-training violation rate at its target across training-set sizes. It is also markedly less conservative than guarding against every mode. Weight-based chance constraints, by contrast, violate their stated risk level as data shrinks. Unimodal Gaussian constraints are both unsafe and conservative. The claim has two falsifiable parts. First, a calibration gap: the realized collision rate of weight-trusting shields exceeds $\delta$ by an amount that grows as forecaster training data shrinks, while the mode-set shield does not. Second, a passive-loosening effect: the mode-set shield's conservativeness drops over an episode and over training even when the robot never nears a constraint.

Relation to Prior Work

The pieces exist separately. Trajectron++ supplies the forecaster: a graph-structured conditional variational autoencoder (a generator with a discrete latent "intent" variable) whose outputs pass through vehicle kinematics so every mode is physically executable. We reuse it and deliberately starve it of data. Recursively Feasible Chance-Constrained MPC Under GMM Uncertainty is the closest work. It plugs Trajectron++ mixtures into model predictive control (repeated short-horizon trajectory optimization) and offers nominal, robust and per-mode contingency planners. Its robust planner is recursively feasible, meaning it never gets stuck without a valid plan, but only under an assumption that forecast uncertainty shrinks monotonically as events approach. That assumption breaks under occlusion or intent reversal. The paper also involves no learning and uses a well-trained predictor.

We differ from it in three ways. We use per-step barrier constraints solved as a small quadratic program rather than full-horizon optimization. We replace mixture weights with calibrated mode membership. We replace the shrinkage assumption with a braking backup check. On the learning side, RL-CBF shields a reinforcement learner with a barrier whose margin comes from Gaussian-process variance on the robot's own dynamics. It trains the policy on the filtered action ("guiding control"), a trick we reuse, but it handles only static constraints. SAPO-RM embeds probabilistic barriers in constrained policy optimization for merging. It forces other vehicles' uncertainty into one Gaussian, which Multiple Futures Prediction and Uncertainty-Aware Driver Trajectory Prediction at Urban Intersections show places mass on the infeasible average path. Probabilistic Enumeration CBFs builds barriers offline around a trained policy in static scenes, so it provides no protection during initial data collection.

The motivating negative result is Breaking Safety Paradox with Feasible Dual Policy Iteration: safer policies produce fewer boundary samples and degrade learned feasibility estimates. Our argument is that this holds when the unknown is the robot's own dynamics or the feasibility boundary itself. It need not hold when the unknown is an external agent the robot can observe from a distance. We test exactly where that argument fails, namely with agents that react to the robot.

Research Program

The system has three parts. The first is a forecaster trained under controlled scarcity. We train Trajectron++ on $N$ logged agent trajectories with $N$ ranging from tens to thousands, and check its per-mode displacement errors and weight calibration on held-out data. The robotics literature on data scarcity is the reason for this regime. RoboCasa and RAD both show that target-domain interaction data stays small and that automated annotators such as hand trackers are noisy. The forecasters a deployed robot actually has will therefore be undertrained.

The second part is the mode-set barrier. At time $t$ the forecaster gives modes $k=1,\dots,K$ with predicted agent positions $\mu_k(t+\tau)$ over a short lookahead $\tau$. We keep a plausible set $\mathcal{M}_t$, the modes whose recent tracking residual is within a calibrated threshold. For each $k \in \mathcal{M}_t$ we define

$$h_k(x,t) = \min_{\tau \in [0,T]} \lVert p(x,t+\tau) - \mu_k(t+\tau) \rVert - r - \rho_k(\tau),$$

where $p$ is the robot's position under a fixed braking rollout from state $x$, $r$ is the combined body radius, $T$ is the braking horizon, and $\rho_k(\tau)$ is a margin set as an empirical quantile of observed deviations from mode $k$ at lookahead $\tau$. These quantiles are updated online in the style of conformal prediction, so the target miss rate is set by the quantile level and not by the mixture weights. The shield solves a quadratic program that minimally alters the learner's action subject to $\dot h_k + \alpha h_k \ge 0$ for all $k \in \mathcal{M}_t$, with $\alpha$ a positive gain. Defining $h_k$ through a braking rollout means feasibility is certified by the existence of the brake itself, not by shrinking uncertainty. This is how we relax the shrinkage assumption of the chance-constrained MPC paper. When the quadratic program is infeasible, the braking backup executes, and we log every such event. Modes leave $\mathcal{M}_t$ only when observed motion contradicts them. Mixture weights are used only to order computation, never to discount safety.

The third part is the learning loop. A policy-gradient learner (PPO) proposes actions, the shield filters them, and the policy is updated on the filtered action as in RL-CBF. We measure three things over training: violations, reward, and a conservativeness index, defined as the fraction of steps where the shield altered the action while the ground-truth future (known in simulation) was in fact safe.

Baselines

The naive reference is an unshielded PPO learner, plus PPO-Lagrangian (a cost-budgeted learner whose penalty weight adapts). On Safety-Gymnasium velocity tasks, Safety-Gymnasium reports that PPO-Lag achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward." The probabilistic-enumeration paper reports a 24.9% real-world collision rate for PPOLag versus 0% with a barrier. We expect both to violate constraints throughout early training. Beating them means a violation count near zero during training at comparable final reward.

The strongest existing competitor is the contingency planner from the chance-constrained MPC paper, fed with the same scarce-data Trajectron++. It is the only method in the file that uses multimodal forecasts for control, and it reports "In all feasible simulation trials, the empirical collision rate was 0%, even with a 5% constraint violation threshold" in CARLA with a well-trained predictor. We run it as a fixed controller, and also as a shield on the same PPO learner, with identical forecasters, horizons and a matched tuning budget over its risk threshold. Beating it means matching its collision rate at full data while holding the target rate at small $N$ where it does not. It also means doing so at lower solve time and without the infeasibility events its robust variant incurs when uncertainty grows.

Two further baselines complete the set. The first is SAPO-RM-style unimodal probabilistic barriers; SAPO-RM reports an "8-meter safety buffer" in merging, and the file gives no collision rate for it. The second is RL-CBF with a worst-case inflated obstacle, where the file reports RL-CBF's 100% training safety only on pendulum and car-following tasks.

The key ablations share the barrier machinery and isolate the ingredient. Ablation (a) uses the same per-mode barriers but applies a weight-based chance constraint that drops modes with weight below $\delta$. Ablation (b) guards every mode for the whole episode with forecaster-given covariances, with neither pruning nor calibration. Ablation (c) is the full method with calibration frozen after the first episode, which separates online passive loosening from a one-time calibration.

Why not just…?

Why not just trust the mixture weights and use a standard chance constraint? It is simpler and is what the closest work does. With abundant data it may be fine, since the MPC paper reports zero collisions. Our hypothesis is specifically that it degrades as $N$ shrinks. Ablation (a) against the full method, swept over $N$, settles this directly. If ablation (a) holds its risk level at small $N$, the calibration gap does not exist and the core claim is falsified.

Why not just guard against every predicted mode, ignoring probabilities? This is the real threat, because it is safe by construction and trivially simple. Its cost is conservativeness. The MPC paper's robust planner already shows higher cost and longer travel times from over-hedging, and exploration suffers when envelopes block narrow high-reward corridors. Ablation (b) measures exactly this. If its reward and conservativeness index are within noise of the full method, the contribution shrinks to margin calibration alone and we would report that honestly.

Why not just learn safety from penalties with PPO-Lagrangian? Penalty methods satisfy constraints only on average and at convergence. The evidence in the file is consistent: Li et al. (paper) describe "safe but useless" versus unsafe penalty regimes, and PPOLag reaches 24.9% collisions on real hardware. The training-curve violation counts in our experiments will show whether this holds with dynamic agents too.

Proposed Experiments

The primary testbed is CARLA unprotected turns and ramp merges with scripted and replayed traffic, which matches the settings of both the MPC and SAPO-RM papers. Here the forecaster is trained on $N \in \{50, 500, 5000\}$ logged trajectories. For a cross-domain condition we also use a forecaster pretrained on nuScenes. The second testbed is Safety-Gymnasium navigation augmented with moving agents that choose between branches at forks, with branch probabilities we control. This lets us plant a rare branch the forecaster underweights. The third is a shared-workspace reaching task in PyBullet, with a Robot Trajectron-style forecaster of human reaching motion (Robot Trajectron, which reports roughly 30 mm average displacement error on Traj100k). This addresses the human-centric constraints Safety-Gymnasium says current benchmarks lack. Metrics are in-training collisions per 1,000 steps, realized miss rate against target, episode reward, the conservativeness index, shield solve time, and backup-trigger rate.

The decisive checks are three. First, plot realized violation rate against $N$ for ablation (a) and the full method; the claim requires the first curve to rise above the target and the second to stay flat. Second, compare reward and conservativeness of the full method against ablation (b) on the fork tasks; the claim requires a clear gain. Third, run the passive-loosening test with reactive agents, whose choice of branch depends on robot proximity. Our safety-paradox argument predicts that loosening stalls for these agents because informative observations then require approaching them. Showing where the advantage disappears is as important as showing where it holds. If loosening does not stall, the argument is incomplete and we would need to explain why.