How Much of a Motion Forecaster's Error Comes From Traffic Regime?

Self-driving systems predict where nearby cars, cyclists and pedestrians will go, and they report a single benchmark score pooled over thousands of scenes. That pooled number hides a question that matters for deployment: is the forecaster equally good in free-flowing midday traffic and in a stop-and-go merge at rush hour, in Phoenix and in Pittsburgh? Or is its accuracy a moving target that shifts with how congested the road is? Medical researchers face the same problem when they combine results from many hospitals. They use random-effects meta-analysis, a statistical method that splits the observed variation in results into two parts. One part is ordinary sampling noise inside each study. The other is real differences between studies, called between-study heterogeneity and written $\tau^2$. We propose to treat each traffic regime as a "study" and run this analysis on motion forecasters.

The core claim is concrete and testable. A meaningful share of a forecaster's regime-to-regime variation in miss rate and calibration is true heterogeneity, not noise, and a large part of it is explained by the time-varying congestion level of the road. Researchers who study temporal accessibility measure this as travel impedance: how slow and how unpredictable the road is right now, as opposed to its fixed geometry. If the claim holds, two practical results follow. First, honest uncertainty for a new city or a new time of day must include $\tau^2$, and pooled calibration will undercover there. Second, the size of the heterogeneity that congestion explains tells us in advance whether feeding live traffic-state information to the forecaster is worth doing. We then test that forecast directly.

Why this gap exists

Leading forecasters on the Waymo Open Motion Dataset (WOMD) condition on agent histories and on static map geometry. MultiPath++ encodes lanes as sparse polylines and uses a gating mechanism to summarize the scene. MTR-A pairs fixed intention queries with refined searching queries over nearby map polylines and outputs Gaussian mixtures. Both pool WOMD's six cities, as described in the WOMD paper, into a single training loss and a single leaderboard number. Neither sees road impedance that changes over time.

Two strands of work motivate looking at congestion specifically. TRACK shows that road-segment representations conditioned on dynamic traffic state beat static ones on city-scale traffic prediction. RSTR shows that ignoring travel-time variance inflates measured accessibility by 24.4% at a 90% confidence level. That second result is a "reliability tax" that appears only once spread, and not just the mean, is taken seriously. Both strands work at the coarse level of 30-minute aggregates or taxi GPS logs, and neither touches the 10 Hz multi-agent interactions where forecasters live.

MetaTra is the closest work on heterogeneity across domains. It uses bi-level meta-learning to improve transfer across pedestrian and sports datasets. However, it improves the mean error and never measures how much performance truly varies across domains, so it constrains our baselines rather than answering our question.

The clinical meta-analyses supply the method. Examples include the epidural hematoma synthesis of 29 cohorts and the vitamin D and tuberculosis analysis. Both use inverse-variance weights $1/(\sigma_i^2+\tau^2)$ and report pooled intervals that widen with $\tau^2$. We reuse that machinery unchanged. The new step is applying it to model evaluation and calibration, not to clinical effects.

Core Claim and Model

We partition held-out scenes into regimes $g$. A regime is a cell crossing a congestion bin with a geographic unit. The congestion bin is computed from the scene itself, using agent density and the median agent speed relative to the lane speed, which is a microscopic proxy for impedance. The geographic unit is a city where scene-level city labels are available. If WOMD does not expose per-scene city, we use map-region clusters; the file does not establish which is available. Argoverse contributes Pittsburgh and Miami (Argoverse).

For each regime we compute a performance quantity $y_g$ for a fixed forecaster. Examples are logit miss rate, the empirical coverage of the forecaster's 90% trajectory region, or the mean negative log-likelihood. We also compute its within-regime sampling variance $\sigma_g^2$ by bootstrapping over scenes. We then fit a random-effects meta-regression:

$$y_g = \beta_0 + \beta_1 x_g + u_g + e_g,\quad u_g \sim \mathcal{N}(0,\tau^2),\quad e_g \sim \mathcal{N}(0,\sigma_g^2),$$

Here $x_g$ is the regime's impedance covariate (its congestion level), $u_g$ is the regime's true deviation from the trend, and $e_g$ is sampling noise. $\tau^2$ is estimated by restricted maximum likelihood.

Two outputs matter. The first is the fraction of heterogeneity that impedance explains, $1-\tau^2_{\text{with }x}/\tau^2_{\text{without }x}$. The second is the prediction interval for an unseen regime, which by analogy with RSTR's effective travel time is an "effective" performance at confidence $\alpha$:

$$y^{\text{new}}_\alpha = \hat\beta_0+\hat\beta_1 x_{\text{new}} + z_\alpha\sqrt{\hat\tau^2 + \mathrm{SE}^2},$$

$\mathrm{SE}$ is the standard error of the fitted trend. The square-root form comes from adding the two independent variance sources. A pooled, fixed-effect analysis sets $\tau^2=0$ and will be too narrow whenever regimes really differ.

Research Program

The first stage is a heterogeneity audit. We train or reproduce an MTR-style forecaster and a MultiPath++-style forecaster on the WOMD training split. On the validation split we compute the regime table with bootstrap variances. We then report Cochran's $Q$, $I^2$ (the share of variation due to real differences), $\tau^2$, and the meta-regression on congestion.

The second stage turns the fitted model into scene-level uncertainty for unseen regimes. For a held-out regime, we scale the forecaster's mixture covariances and mode-probability temperature so that the predicted regime-level coverage matches the random-effects prediction interval. The scaling is fit only on the remaining regimes.

The third stage is the intervention the audit points to. We add a traffic-state input to the forecaster, computed from observed history only so that no future information leaks into the input. The input is a scene-level density and speed token, plus, where map matching allows, a TRACK-style segment embedding. We then check two things: whether miss rate improves, and whether residual $\tau^2$ shrinks by roughly the amount the meta-regression predicted.

The audit is useful even if the intervention fails. A large $\tau^2$ that congestion does not explain is itself a finding. It would mean the heterogeneity comes from something else, such as city-specific driving culture, which points to a different fix.

Baselines

The naive reference point is the pooled, fixed-effect view used on leaderboards: one global miss rate, with calibration fit on the pooled validation set. As a variance estimate it is the same meta-analytic machinery with $\tau^2$ forced to zero. This doubles as the key ablation. It uses the same data, regimes and bootstrap, so any coverage gap between it and our method is attributable to modeling $\tau^2$ alone.

The strongest existing forecaster is MTR-A, which reports Soft mAP 0.4594 and Miss Rate 0.1160 on WOMD (MTR-A). MultiPath++ reports mAP 0.409 and minADE 0.556 m (MultiPath++). We found no reported per-regime or per-city breakdown of either on Lacuna, so the audit itself produces the first such numbers. Our reproductions must land near these pooled figures before regime results are trusted. We will use the same training budget for the conditioned and unconditioned variants. Beating this baseline means one of two things. Either the impedance-conditioned model lowers pooled miss rate while also lowering residual $\tau^2$, or, with the forecaster fixed, our intervals achieve near-nominal coverage on held-out regimes where pooled calibration undercovers.

The strongest calibration competitor is group-split conformal prediction. It holds out whole regimes, treats them as exchangeable units, and sets interval widths from the spread of regime-level scores. This is the nonparametric cousin of our method, so it gets the same regimes and the same leave-one-regime-out protocol. We beat it if we match its coverage with narrower intervals by using the congestion covariate, especially for regimes at congestion levels poorly represented in training.

MetaTra-style meta-learning (MetaTra) is the domain-generalization baseline. We give it the same compute budget and compare its mean error and its residual $\tau^2$.

A second ablation fits the random-effects model without the congestion covariate. It isolates whether impedance, rather than generic grouping, carries the explanatory power.

Why not just…?

Why not just report the forecaster's metrics per congestion bin? Raw per-bin numbers mix sampling noise with real differences, so small bins look spuriously extreme and large bins look spuriously stable. Telling these apart is exactly what $\sigma_g^2$ versus $\tau^2$ does. The clinical syntheses show the consequence: with many small cohorts, the fixed-effect intervals and the honest intervals diverge. Per-bin tables also give no interval for a regime that has not been observed, which is the deployment question.

Why not just use conformal prediction per regime? Per-regime conformal needs labeled data from the target regime, which a new city does not have. Pooled conformal assumes scenes from all regimes are exchangeable, which the audit tests directly. Group-split conformal is a real threat to the interval-construction part of this work, and we run it as a baseline. If it matches our coverage at equal width, we will say so. The contribution then narrows to the audit and the meta-regression, which conformal methods do not provide: they say how wide the intervals must be, not why the regimes differ.

Why not just add traffic-state features to the forecaster and skip the statistics? We do add them, in the third stage. The meta-analysis is the instrument that tells us whether that change addresses regime heterogeneity or only nudges the average. Without it, a small pooled gain could hide a large improvement in congested regimes, or none at all, and the leaderboard number would not distinguish these cases.

Why not just use meta-learning like MetaTra? Meta-learning can improve transfer accuracy, but it outputs a model, not a measured spread of performance across domains. We will check whether it reduces $\tau^2$ or only shifts the mean. If it eliminates heterogeneity at equal compute, that is a strong negative result for the conditioning route, and we will report it.

Proposed Experiments

The first decisive check is the size of the heterogeneity. On WOMD validation regimes, we expect $I^2$ for regime-level miss rate to be well above zero. If it is near zero, the proposal's premise fails and we stop at that finding.

The second check is the coverage test. We hold out each congestion bin in turn, and each city or map-region cluster in turn. We compare how often the regime's true miss rate and scene-level 90% trajectory coverage fall inside the interval under three methods: fixed-effect, random-effects, and group-split conformal. Fixed-effect intervals should undercover in the extreme congestion bins.

The third check is the prediction-then-intervention test. Before training the conditioned forecaster, we record the share of $\tau^2$ that congestion explains. Afterward, we measure how much $\tau^2$ actually drops. Agreement between the two would show that the audit can guide architecture decisions.

The fourth check is transfer across datasets. We fit the meta-regression on WOMD regimes and ask whether its prediction interval covers the forecaster's regime-level performance on Argoverse's Pittsburgh and Miami scenes. Argoverse's own reported baselines, ADE 2.25 and FDE 4.67 at 3 s for LSTM ED+social+map, serve only as a sanity anchor for the evaluation pipeline.

The main risks are that the in-scene congestion proxy is too noisy to separate regimes, and that city labels are unavailable. Either would show up as wide bootstrap variances, and the analysis will report it rather than hide it.