Pruning Interaction Futures with a Measured Miss Rate

A self-driving planner cannot afford to reason about every way the surrounding traffic might unfold. It keeps a handful of likely joint futures, meaning combinations of what each nearby car and pedestrian does, and drops the rest. That dropping step, called probabilistic pruning, is where safety is quietly decided. If the predictor that assigns the probabilities is overconfident, the pruned set is too small. If the other road users react to the ego vehicle's own maneuver, which marginal predictors ignore, the pruned set is wrong. And if the planner is only ever tested in simulators whose agents are polite or were built from the same model the planner uses, nobody notices. We propose to prune with a guarantee attached to the pruning step itself. The planner keeps the smallest set of joint futures whose chance of missing what actually happens is at most a chosen level $\alpha$, conditioned on the ego plan being considered. That level is set by conformal calibration on real driving logs, a distribution-free method that turns held-out errors into a finite-sample coverage bound. We then check whether the bound still holds when the planner is driven in closed loop against simulated agents from families other than its own predictor.

The contribution has two parts that only make sense together. The first is a pruning rule whose failure rate is a measured, tunable quantity rather than a side effect of a probability threshold. The second is an evaluation protocol that treats simulation as evidence about that failure rate. Its central measurement is how far the in-simulation miss rate drifts from the rate calibrated on logs as the simulated agents become less like the planner's own world model. That drift is a direct, planner-relevant measurement of the reality gap for interaction reasoning.

Core Claim

Existing probabilistic pruning in interactive planning thresholds per-mode probabilities that come from exogenous, non-reactive obstacle models. Bethge et al. drop a driver-intention mode from their Gaussian-process MPC once its posterior falls below $\epsilon_P$. They test this only on pairwise encounters and explicitly leave multi-agent scaling open. SH-MPC removes outlier obstacle scenarios while bounding joint collision probability, but it samples those scenarios from synthetic Gaussian or mixture distributions that do not respond to the ego vehicle, and it is validated in a custom eight-pedestrian simulation. Both rules rest on predicted probabilities being calibrated. CCTR shows that assumption fails for learned trajectory predictors. It reports near-zero correlation between predicted variance and error before its fix, and 11 to 22 percent calibration-error reductions after it.

Our claim is that calibration should target the event a pruning rule actually cares about, which is whether the realized joint future lies outside the retained set. It should also be conditioned on the ego plan. Doing so gives a coverage statement that per-mode temperature scaling does not give. The hypothesis to test is that this statement degrades in a predictable way under the simulator mismatch documented by the Waymo Open Sim Agents Challenge (logged-human realism 0.722 versus about 0.645 for the best learned sim agents) and by the nuPlan authors, who describe their IDM reactive agents (Intelligent Driver Model, a rule-based car-following controller) as simplistic and polite.

Research Program

The predictor is a joint, action-conditional interaction model. It takes the scene $s$ and a candidate ego trajectory $a$ and returns $K$ joint modes $m_1,\dots,m_K$ with probabilities $p(m_k \mid s, a)$, following the action-conditional formulation of Banijamali et al.. We build it by injecting the ego trajectory as a fixed query into the joint decoder of BeTopNet, using its public code at OpenDriveLab/BeTop. BeTopNet contributes the braid-style "who yields to whom" topology. That topology gives a natural definition of when two joint futures are the same for planning purposes: they are the same if they share a crossing structure and lie within a distance $\delta$ of each other.

The pruning rule sorts modes by probability and keeps the shortest prefix whose cumulative probability exceeds a threshold $\lambda$. We choose $\lambda$ by split conformal calibration on held-out logged scenes, conditioning on the logged ego trajectory. For each calibration scene we record the smallest $\lambda$ at which the realized future is covered. We then set $\hat\lambda$ to the appropriate empirical quantile, so that

$$\Pr\big[\, y \notin R_{\hat\lambda}(s,a) \,\big] \le \alpha ,$$

where $y$ is the realized joint future, $R_{\hat\lambda}(s,a)$ is the retained mode set, and the bound holds on average over scenes exchangeable with the calibration set. The quantity on the left is the miss rate, and it is the object the whole project measures.

The retained modes feed a contingency planner in the style of LookOut. It commits to a shared first segment and branches into one sub-plan per retained mode, with scenario constraints linearized into safe polytopes as in SH-MPC. The retained set now varies from step to step, so the planner spends branches where the interaction is uncertain and saves them where it is not. LookOut instead fixes the budget at $K=15$.

The honest weak point is counterfactual coverage. Logs show only the reaction to the ego trajectory that was actually driven, so the guarantee is formally valid only for plans near the logged one. We do not paper over this. We report the log-calibrated miss rate stratified by how far the candidate ego plan deviates from the logged plan. We then use closed-loop simulation as the only available proxy for larger deviations, which is exactly where the third direction enters.

Simulation is used as a graded ladder of agent families, each more distant from the planner's predictor than the last. The nearest rung is self-play, where the planner's own predictor drives the other agents; we expect this rung to be optimistic. The next rung is a learned reactive sim-agent policy trained under the WOSAC protocol on the Waymo Open Motion Dataset (WOMD). It is a different model family from the planner's predictor, for example a promptable reactive policy like ProSim. After that comes nuPlan's IDM agents. The farthest rung is an "unyielding" IDM variant with reduced yielding and shorter gaps, in the spirit of the targeted stress tests of TRAVL. TRAVL showed that nominal-traffic training transfers poorly to such tests (69.6 versus 90.3 percent pass rate).

On every rung we log the miss rate, collision rate, and progress. The resulting miss-rate-versus-mismatch curve is the deliverable that tells a practitioner how much to widen $\alpha$ before trusting a simulated safety number.

Baselines

The naive reference is no pruning. The planner branches on all $K$ predicted modes under the same wall-clock budget and truncates the solver when the deadline hits. This shows whether pruning buys anything or merely hides modes that a full planner could have handled. A second naive point is the nuPlan rule-based IDM planner. The nuPlan paper reports a closed-loop reactive score of 0.76 for it, against 0.67 for UrbanDriver, and this number anchors what a learning-free controller achieves.

The strongest existing methods fall into two groups. The first is pruning by a fixed probability threshold on raw predictor outputs, which is the mechanism of Bethge et al. and the scenario-removal step of SH-MPC. We run both on our predictor and planner so that only the pruning rule differs. SH-MPC reports no closed-loop benchmark number; no reported result found on Lacuna for SH-MPC on nuPlan or WOMD. The second group is published interactive planners on nuPlan. GameFormer reports a nuPlan planning score of 0.8288. The nuPlan paper reports 0.895 for the top hybrid challenge entry (CS Tu), which pairs learned proposals with rule-based safety checks. BeTopNet reports relative gains (+3.8 percent on Test14-Inter, +6.2 percent non-reactive closed loop) but no absolute score in the file. We rerun BeTopNet from its public code on our splits rather than quoting a number.

All learned components share the same training data, the WOMD and nuPlan training splits. They receive the same hyperparameter search budget and run under the same per-step planning deadline at 10 Hz on the same GPU. Beating the strongest competitor means one of two outcomes on Test14-Inter and the unyielding-IDM rung. The first is a matching closed-loop score with a lower collision rate and a miss rate within a stated tolerance of $\alpha$. The second is a matching collision rate at a higher score, meaning more progress and less freezing.

Three ablations isolate the ingredients. Replacing conformal selection of $\lambda$ with CCTR-style temperature scaling followed by a fixed threshold isolates set-level calibration. Replacing the action-conditional predictor with its marginal counterpart, which ignores the ego plan, isolates reciprocity. Fixing the retained set size at its average value isolates whether adaptivity, rather than a better average budget, drives any gain.

Why not just…?

Why not just calibrate the predictor with temperature scaling and keep a fixed threshold? This is the most serious threat. CCTR shows post-hoc calibration substantially improves downstream planning error (17 to 46 percent on Argoverse). If it also makes the pruned set's miss rate hit $\alpha$, conformal selection adds nothing. The distinction is that temperature scaling matches average per-mode confidence, whereas the pruning rule needs coverage of a set conditioned on the ego plan. Nothing forces the first to deliver the second, especially given CCTR's own finding that calibration error varies with proximity to the ego vehicle. The first ablation settles this directly. If the two rules produce indistinguishable miss-rate curves across the simulation ladder, we will report that and recommend the simpler method.

Why not just keep a fixed top-K, as LookOut does? LookOut reports 8 percent fewer collisions and a 21 percent gain in progress per collision with only 15 samples. A fixed budget is clearly workable. It fixes compute but leaves the miss rate uncontrolled, so the miss rate rises in exactly the dense, high-entropy scenes that matter, and budget is wasted in quiet scenes. The fixed-size ablation, stratified by scene entropy on Test14-Inter, shows whether adaptivity matters. If fixed-$K$ matches our miss rate in the dense stratum, the adaptive component should be dropped.

Why not just put a rule-based safety layer on top of any learned planner? The hybrid CS Tu entry scoring 0.895 on nuPlan is strong evidence that this works against IDM agents. Chen et al. argue that reactive safety controllers cannot proactively negotiate merges and dense intersections, and the nuPlan authors concede that IDM agents are polite. This means the high score may reflect the benchmark rather than the method. We therefore include a hybrid baseline with a time-to-collision safety filter and compare it on the unyielding rung, where politeness can no longer be exploited. A hybrid that stays ahead there would be a real negative result for this proposal.

Why not just evaluate on log playback, since logged humans are real? Log playback is non-reactive, and WOSAC documents that even centimetre deviations produce pose-divergence collisions. That makes it useless for testing reciprocal pruning in closed loop. We use logs only for the part they are valid for, which is calibration and miss-rate measurement near the factual ego trajectory.

Proposed Experiments

The first experiment is open-loop and runs on WOMD interaction scenes and nuPlan logs. Against the logged ego trajectory, we measure the miss rate of each pruning rule at $\alpha \in \{0.01, 0.05, 0.1\}$ together with the average retained set size. We stratify by deviation of the candidate plan from the log and by scene density. We also adapt the fatality-aware decomposition of the NGSIM reaction benchmark, which separates false alarms (conservatism) from missed dangerous reactions (non-defensiveness) and weights them by inverse time-to-collision. With it we report whether misses concentrate in dangerous reactions. That paper found neural predictors had lower average error but roughly twice the non-defensiveness of inverse reinforcement learning (0.1303 versus 0.0698), which is exactly the failure a miss-rate bound should expose.

The second experiment is closed-loop. It runs on nuPlan's standard and Test14-Inter splits across the four-rung agent ladder and reports the nuPlan score, collision rate, a freezing indicator (zero progress for more than a fixed interval), solver time, and in-simulation miss rate.

Two decisive checks follow. The first is whether the log-calibrated $\alpha$ is honored under self-play but violated under IDM and unyielding agents. That would confirm that simulated safety is inflated when agents share the planner's model, and it would give the size of the correction. The second is whether conformal pruning beats temperature-scaled thresholding on miss rate at equal compute on the unyielding rung. If neither check comes out as predicted, the project has still produced a reusable measurement, the miss-rate-versus-agent-mismatch curve for probabilistic pruning, which no paper in this literature currently reports.