Stratum-Calibrated Interaction Forecasts

Driving datasets are samples of convenience. Most recorded vehicle-hours are steady cruising, a few percent are merges, cut-ins and yield-or-go conflicts, and the mix depends on where and when someone parked a drone or drove a sensor car. A forecaster's benchmark score is an average over that accidental mix. So is its apparent calibration, meaning whether an event it calls 30% likely happens about 30% of the time. Survey statisticians have long handled exactly this situation: estimating a quantity for a target population from a skewed sample. Their main tool is post-stratification, which cuts the data into cells, estimates within each cell, and reassembles the cells using target-population proportions. When cells are sparse, they use multilevel regression, which lets thin cells borrow strength from related ones.

This proposal adapts that toolkit to one concrete object: the probability a multi-agent forecaster assigns to a discrete interaction outcome, such as whether a merging car yields or proceeds, or whether an agent crosses a conflict point before the ego vehicle. We fit a small hierarchical logistic recalibration layer on top of any trained forecaster, with one intercept and one slope per interaction stratum, partially pooled across road topology, agent type and criticality. We then post-stratify the per-stratum errors to estimate how the forecaster would perform under a different traffic mix. The central claim is that calibration conditional on an input-defined stratum carries over to any new mix of strata, while aggregate calibration does not. Pooled per-stratum recalibration should therefore beat both global recalibration and reweighted training at predicting and achieving calibrated performance in a shifted target population, and it costs almost nothing to fit.

Core Claim and Why It Is the Right Combination

The obvious idea at this intersection is to reweight or oversample rare interactive scenarios during training. The research file finds no paper doing this for trajectory models. Still, a reviewer would rightly call it ordinary long-tail rebalancing, and the statistics argue against it as the main lever. If strata are defined from the observed history, which is required to avoid leaking the future, then changing the stratum mix is covariate shift. The best conditional prediction inside each stratum does not change. Reweighting at training time only helps through capacity or misspecification effects, and it pays in variance. Overlap, matching, or entropy weights: what are we weighting for? documents extreme-weight variance for inverse-probability schemes. Correcting Annotator Bias in Training Data: Population-Aligned Instance Replication (PAIR) avoided loss weighting as unstable. What does move under a change of mix is the aggregate: overall calibration and overall benchmark scores.

The right place for survey machinery is therefore twofold. First, make the forecaster calibrated within each stratum. Second, report and select models using post-stratified estimates for an explicitly stated target mix. PAIR supplies the motivating evidence: in text toxicity classification, skewed annotator pools left F1 flat but sharply increased calibration bias. The proposal differs from PAIR in three ways. It works on dynamic interaction strata rather than demographic ones. It corrects by post-hoc hierarchical recalibration rather than instance replication. It adds an estimate of performance under a target population.

Mechanism

Take a trained forecaster that outputs, for an agent pair in stratum $s$, a probability $p$ of a discrete interaction outcome $y$ (for example, yield). A stratum is the crossing of four factors, all computed from the observed history and map: scenario type, agent type, speed band, and inverse time-to-collision band at prediction time. The recalibrated probability $q$ is

$$\operatorname{logit} q = a_s + b_s \operatorname{logit} p,$$

where $a_s$ and $b_s$ are sums of per-factor random effects with shared Gaussian priors. Sparse cells therefore shrink toward their neighbours in road topology or agent type instead of being fit on their own. This is multilevel regression in the sense used by MRP studies such as Swaying the Public?, applied to a forecaster's logit rather than to a survey response.

For a target traffic population with stratum shares $\pi_s$, the post-stratified estimate of any per-example metric $m$ (Brier score, miss rate, fatality-weighted error) is

$$\hat M_{\text{target}} = \sum_s \pi_s \, \hat m_s,$$

where $\hat m_s$ is the multilevel model's estimate of the stratum-mean metric. We obtain the shares $\pi_s$ from a target dataset's unlabeled histories, or from a stated exposure mix. Unlabeled histories suffice because strata depend only on inputs.

Two boundaries are part of the design. Strata with no source data at all cannot be recovered; the file calls this the "reweighting the willing" limit (Pluralistic Preference Alignment via Sortition-Weighted RLHF). Pooling only extrapolates along the factors we crossed, so we report posterior width per cell and flag cells the target needs but the source lacks. Strata are also imperfect proxies for latent intent. We test sensitivity by substituting behaviour clusters from the domain-guided CVQ-VAE of Behavior-Centric Extraction of Scenarios from Highway Traffic Data. That work extracts and clusters interactive highway scenarios but never uses the clusters to reweight or calibrate a predictor. Its clusters must be computed on history only for our use.

Research Program

The interaction outcome follows the prototype-trajectory discretization of Towards a Fatality-Aware Benchmark of Probabilistic Reaction Prediction in Highly Interactive Driving Scenarios. That benchmark turns NGSIM ramp merges into discrete yield/pass patterns and scores them with a Brier score weighted by inverse time-to-collision, decomposed into conservatism (false alarms) and non-defensiveness (missed risks). It weights only at evaluation, in one scenario, and without correcting for the sample mix. We keep its decomposition as a metric and add stratified calibration and population-level estimation. For urban data we derive the same binary conflict-order event from the ranked trajectory modes of a MultiPath++-style forecaster. MultiPath++ fuses sparse polyline map and agent encodings and outputs weighted trajectory modes. We sum the mode probabilities on each side of the conflict point, so the recalibration layer attaches to an existing strong forecaster without retraining it.

The work proceeds in three stages. We first build history-only stratifiers for highD, NGSIM, the Waymo Open Motion Dataset and Argoverse. We then fit the pooled recalibration on source validation splits. Finally, we measure two things: whether post-stratified estimates predict the target's measured metrics, and whether recalibrated probabilities are calibrated within each stratum and in aggregate on the target.

Baselines

The naive reference is the raw forecaster, unrecalibrated, with its target performance "estimated" by its source validation score. This is what current benchmarking implicitly does.

The strongest cheap competitor is global temperature or Platt scaling: one or two parameters for all strata, fit on the same source validation split. Beating it means lower per-stratum and aggregate expected calibration error on the target, and a lower fatality-weighted score. The benchmark to target is IRL's fatality-weighted Brier score of 0.2053 on NGSIM merges, against 0.2361 for the mixture density network and 0.2551 for the hidden Markov model (Towards a Fatality-Aware Benchmark). We will count it a success if a recalibrated mixture density network, which already has the best plain Brier score there (0.1099), matches or beats 0.2053 while keeping plain Brier near 0.11. Its non-defensiveness should fall toward IRL's 0.0698 from its own 0.1303.

The strongest training-side competitors are PAIR-style replication and inverse-probability-weighted retraining to the target stratum mix. Both get the same data, the same architecture, and a matched tuning budget of epochs and hyperparameter trials. A second training-side competitor is cost-sensitive training with inverse time-to-collision weights, which is the most direct weighted-logistic reading of the fatality-aware benchmark.

For urban data the anchor is MultiPath++'s reported Waymo Open Motion Dataset mAP of 0.409 and minADE of 0.556 m. Recalibration only changes the event-probability readout, so it must leave these figures unchanged. For calibration error on this binary event, no reported result was found on Lacuna, so we will establish it ourselves.

Two ablations isolate the key ingredient, partial pooling. The first fits unpooled per-stratum Platt scaling, which exposes the variance cost in thin cells. The second fits pooled recalibration over randomly permuted strata of matched sizes, which tests whether interaction structure matters or just extra parameters.

Why not just…?

Why not just use global temperature scaling? If miscalibration is roughly uniform across strata, one temperature fixes it, and this proposal's calibration half is unnecessary. We do not assume otherwise. The mixture density network's pattern in the fatality-aware benchmark, good average Brier with poor non-defensiveness, suggests the errors are concentrated in critical strata, but that is one scenario. The per-stratum calibration comparison on the target settles it. If global scaling ties, we report that, and the post-stratified evaluation half stands on its own.

Why not just oversample or reweight rare interactions during training? Under covariate shift this changes the conditional only through model capacity, and it adds variance. PAIR avoided loss weighting for this reason, and the sortition study found hard quota panels beat soft inclusion-probability weighting, with a pairwise win probability of 0.618 against the full convenience pool (Pluralistic Preference Alignment via Sortition-Weighted RLHF). Retraining also needs one run per target mix; recalibration needs one small fit. This alternative is a real threat on small-capacity models, and the matched-budget retraining baseline decides it.

Why not just train with criticality weights, as a cost-sensitive logistic loss? Weighting the likelihood by inverse time-to-collision deliberately distorts probabilities toward caution. That blurs the probability estimate and the decision into one. We keep probabilities calibrated and apply the criticality cost at decision time. If criticality-weighted training wins on the fatality-weighted score but loses on plain calibration, that confirms the separation matters.

Why not rely on attention-based agent importance to focus on what matters? Importance is in your attention reads agent importance from cross-attention and reports a 0.477 correlation with leave-one-out trajectory change on nuPlan. That picks which agents to attend to, but it does not make the forecaster's probabilities trustworthy for any population. We will still test attention-importance bands as an alternative stratifying factor.

Proposed Experiments

The decisive test is cross-population transfer with ground truth held back. We fit recalibration and the multilevel error model on highD. We compute stratum shares from NGSIM histories alone, then compare the post-stratified estimates of Brier score, non-defensiveness and conservatism against the metrics measured on NGSIM labels. We repeat the test from Argoverse to the Waymo Open Motion Dataset. The claim holds if post-stratified estimates track measured target metrics substantially better than raw source scores. It also requires that model rankings which flip between source and target be predicted by the estimate.

A controlled second test builds synthetic target mixes by subsampling held-out Waymo data at interaction-heavy ratios. This lets us chart estimate error against how extreme the shift is and how sparse the cells are. We expect unpooled estimates to degrade fastest in thin cells, since their variance grows as one over the cell count, while pooled estimates degrade gracefully.

The third test measures per-stratum calibration after each baseline, and the fatality-weighted Brier score on the NGSIM merge benchmark against the IRL figure. We will report cells absent from the source separately, rather than folding them into averages.