Phantom Hazards: Separating "What Others Will Do" from "What Is Out There" in Traffic Prediction

When the car two vehicles ahead brakes hard, a driver reasonably infers that something is in the road. That inference is sound for the first observer. It becomes unreliable when the third car brakes only because the second did, and the fourth only because the third did. Economists call this an information cascade: each agent's action reflects copying rather than its own evidence, so a long chain of identical actions carries little more information than the first one. Learned motion forecasters for autonomous driving, the networks that predict where surrounding vehicles will go, read neighboring vehicles as parallel sources of context through attention or pooling. Nothing in their design asks whether five braking cars are five independent witnesses or one witness and four copiers. A planner built on such a forecaster can be pulled into a "phantom hazard," braking or swerving for an obstacle that only the first car thought it saw, or that no car saw at all.

The proposal has three parts. First, we measure whether current forecasters herd in this way, using counterfactual chain-braking scenes generated inside a closed-loop traffic simulator. Second, we propose a model that keeps two quantities apart. The first is a behavior forecast, which should predict that human followers will herd, because they do. The second is a latent hazard belief, which should not herd: when a vehicle's maneuver is already explained by its own predecessors, its weight as evidence about the road is discounted. Third, we test whether a planner that uses the hazard belief can avoid needless braking without missing real obstacles.

Core Claim

Current forecasters merge two different questions: what the other agents will do, and what state of the world their actions reveal. Cascade theory says these questions come apart exactly in the chain-like situations traffic produces. Once an agent copies a predecessor, its action says nothing new about the hidden state (Chen et al.; Huttenlocher et al.; Theodoropoulos et al.). We claim that a forecaster with an explicit hazard variable, whose evidence from each neighbor is weighted by how much of that neighbor's maneuver its predecessors fail to explain, will produce calibrated hazard beliefs under chain braking. Its behavior forecasts will be no worse than a strong transformer forecaster's on standard metrics. A model that only adds visibility information, or only trains on more chain-braking data, will not generalize across chain lengths and layouts that are absent from training.

The weighting rule fits in one line. For neighbor $i$ with observed maneuver $m_i$, we define its residual surprise as

$$s_i = -\log p_\theta\left(m_i \mid m_{\mathrm{pred}(i)}, v_i\right),$$

where $m_{\mathrm{pred}(i)}$ are the maneuvers of the vehicles $i$ could see ahead of it, $v_i$ is the scene region visible to $i$ but not to the target, and $p_\theta$ is a learned per-agent reaction model. A maneuver that the vehicle's predecessors fully predict has low surprise. It enters the hazard belief with low weight, while still entering the behavior forecast at full strength. This is "explaining away" carried across a queue, and it generalizes the binary sequential-learning updates of the cascade literature to continuous trajectories.

How the Prior Work Shapes This

The cascade literature supplies both the motivation and the stylized theory. Bandit Social Learning under Myopic Behavior proves that greedy or pessimistic populations abandon a good option forever after unlucky early evidence. Pessimism Traps and Algorithmic Interventions models such traps as cascades and derives closed-form ideal behavior for binary signals with accuracy $p$. Ho et al. show that contaminated observations leave a permanent error floor for self-interested learners. All of this work is binary and single-step. We do not reuse its interventions. We use its ideal Bayesian posterior as a ground-truth target in a stylized platoon, which gives a calibration check that real driving data cannot.

The interaction-modeling literature comes closest without crossing over. PiP conditions neighbor forecasts on the ego vehicle's plan. Prediction by Anticipation factors the scene into the ego action followed by a learned environmental reaction. Both model one hop of influence, from ego to neighbor, and neither represents a hidden hazard that a chain of reactions might reveal or fake. LookOut samples joint futures that call for different ego reactions and plans contingently, but it treats surrounding agents as draws from a fixed generator rather than as noisy reporters of the road. ISAP separates epistemic uncertainty about social context from uncertainty about the map and the agent's own motion. Its social branch measures whether the scene is unfamiliar, not whether the testimony is redundant, so a familiar-looking cascade would leave it confident. The transformer forecasters MTR-A and MultiPath++ define the accuracy bar our behavior head must match.

The fatality-aware benchmark of Towards a Fatality-Aware Benchmark supplies our evaluation lens. It splits error into conservatism (false alarms) and non-defensiveness (missed threats). That split maps directly onto phantom braking and missed real hazards.

Research Program

The work proceeds in three stages. The first stage is a diagnostic benchmark. Starting from Waymo Open Motion Dataset scenes (WOMD) that contain queues of three to eight vehicles, we use the promptable simulator ProSim to generate counterfactual rollouts in which the lead vehicle receives an action prompt to brake hard. ProSim's action tags drive closed-loop rollouts with an average displacement error of 0.28 m when all prompt types are combined. Each rollout falls into one of three regimes. In the real-hazard regime, a static obstacle is placed in the lane. In the phantom regime, no obstacle exists. In the occluded-real regime, an obstacle is visible only to the lead car. Followers react through the simulator's reactive policies, which gives naturally herding human-like followers.

We then probe the frozen forecasters. We ask whether the predicted trajectory of the last vehicle, and the planner's response, differ between the phantom and real regimes when that vehicle's own line of sight is clear. A forecaster that cannot tell them apart is herding. One caveat matters: simulated followers inherit ProSim's learned behavior. We therefore also mine naturalistic chain-braking events from WOMD and NGSIM to check that the behavior statistics are plausible.

The second stage builds the model. A transformer scene encoder in the style of MTR-A feeds two heads. The behavior head is trained on standard trajectory losses. The hazard head outputs a per-lane-segment occupancy belief and aggregates neighbor evidence weighted by the residual surprise $s_i$. The reaction model $p_\theta$ is trained on naturalistic data to predict each vehicle's maneuver from its visible predecessors. Visibility regions come from map geometry and agent boxes. The hazard head is supervised with simulator ground truth in counterfactual scenes, and with weak labels in real data where a stopped object later appears in the map region.

The third stage is closed-loop use. A contingency planner in the style of LookOut consumes either the hazard belief or the raw behavior forecast as its hazard input. We measure whether it avoids phantom stops without becoming non-defensive.

Baselines

The naive reference point is a constant-velocity forecaster paired with a planner that brakes whenever any visible vehicle ahead decelerates past a threshold. This is the herding policy in its purest form. No reported result was found on Lacuna for this baseline on WOMD. Any proposed model must cut its phantom stops sharply while keeping its real-hazard recall.

The strongest existing forecaster is MTR-A, which reports Soft mAP 0.4594 and Miss Rate 0.1160 on the 2022 Waymo motion prediction challenge. We retrain it on the same WOMD splits, with the same compute budget and hyperparameter search as our model. We then read its implied hazard response from its forecast of the target and the planner's reaction to it. Beating it means matching its Soft mAP within noise on the standard split while showing a significantly smaller phantom-versus-real gap in false alarms and an equal or lower miss rate on real and occluded-real hazards.

ISAP is the strongest uncertainty baseline. It reports an out-of-distribution AUROC of 0.838 against 0.553 for a 10-model ensemble on nuScenes map shift. We use its social-context uncertainty to gate the planner's hazard response, under the same training data. Beating it means better discrimination between phantom and real hazards, measured by AUROC of the hazard score across regimes.

Three ablations isolate the key ingredient. The first gives the hazard head all neighbors' maneuvers but weights every neighbor equally, which is independent evidence counting. The second adds visibility masks but no residual-surprise weighting. The third fine-tunes MTR-A on counterfactual chain-braking data with no structural change. All three use the same data and the same tuning budget as the full model.

Why not just…?

Why not just train a bigger joint forecaster on more data, so attention learns redundancy on its own? This is a real threat. Attention can in principle represent explaining-away. However, naturalistic logs almost never contain phantom chains with known ground truth, so nothing in the training signal rewards the distinction. Our third ablation tests this directly, and we hold out chain lengths and road layouts at test time. If the fine-tuned MTR-A generalizes to held-out structures as well as the full model does, the structural claim fails, and we will report it as a data result rather than a modeling result.

Why not just use an uncertainty estimator such as ISAP or a deep ensemble? These estimators flag scenes that are unfamiliar. A cascade looks entirely ordinary: every follower behaves plausibly given its predecessor. We therefore expect ISAP's social uncertainty to stay low in phantom scenes, and the ISAP baseline measures exactly this. The 0.553 AUROC reported for the ensemble in ISAP suggests ensembles are a weaker option still.

Why not just mask each neighbor by what it could see that the target cannot? Visibility is half of the idea, and it may deliver most of the gain in simple single-lane queues. Visibility alone cannot tell a lead car that reacted to a real occluded obstacle from one that braked idiosyncratically. In both cases the lead car sees a region the follower does not. The second ablation measures how much residual-surprise weighting adds over visibility. We expect the difference to grow with chain length and multi-lane topology. If it does not, we will recommend the simpler masked model.

Proposed Experiments

The first decisive check is the herding diagnostic. On ProSim counterfactuals, we plot each forecaster's false-alarm rate for the clear-view last vehicle against chain length, separately for phantom and real regimes. Herding appears as phantom false alarms rising with chain length. The ideal response is flat in the phantom regime.

The second check is calibration against theory. In a stylized platoon whose hazard and reaction signals follow the binary model of Pessimism Traps and Algorithmic Interventions, with $p$ swept from 0.6 upward, we compare each model's hazard belief to the exact Bayesian posterior. We measure this with expected calibration error.

The third check guards against regressions. We report Soft mAP, mAP, minADE, and miss rate on the standard WOMD interactive split, and on Argoverse 2 (Argoverse 2) for transfer. These must stay level with MTR-A and MultiPath++, which reports mAP 0.409 on WOMD.

The fourth check is closed-loop planning. A LookOut-style planner runs with each hazard source on ProSim rollouts. We score conservatism and non-defensiveness using the inverse-time-to-collision weighting of Towards a Fatality-Aware Benchmark, where IRL reached $C = 0.0178$ and $D = 0.0698$ against $0.0493$ and $0.1303$ for a mixture density network in pairwise merging. We also report progress per collision.

The proposal succeeds if three things hold together. Phantom-regime conservatism must fall below MTR-A's. Occluded-real non-defensiveness must not rise. And the advantage over the visibility-only ablation must grow with chain length.