Show Only the Hazards the Driver Will Miss: Conflict-Selected AR Overlays for Occluded Road Users

A windshield display that overlays graphics on the real road, called augmented reality (AR), can show a driver things the driver cannot see: a pedestrian hidden behind a delivery truck, or a car about to pull out of a blind driveway. The models that could supply this information already exist. Trajectory forecasters predict several distinct possible future paths for each road user, which is what "multi-modal" means here. Occlusion-reasoning models infer road users hidden from the car's sensors. Vehicle-to-vehicle sharing lets one car borrow another car's view. The display is the weak link. A driving-simulator study found that highlighting every object in dense traffic reduced driver situation awareness, and that only highlights on peripheral objects in light traffic helped (Gao et al.). Occlusion models also tend to over-predict hidden agents when visibility is poor (Scene Informer). Feeding these models' raw output to the windshield would produce exactly the clutter that hurts drivers.

This note proposes treating the overlay as a selection problem. At each moment the display should render only the few road users, visible or inferred, that are likely to cross the driver's own planned path and that the driver is unlikely to have noticed. The forecaster's multiple futures are not drawn on screen. Their only job is to compute a calibrated probability of conflict. Occlusion is the case where this matters most, because a hidden agent has no observed velocity and the driver has no way to have seen it.

Core Claim

Our claim is that a display policy that ranks candidate overlays by expected benefit, with forecasts that include inferred hidden agents and a hard cap on how many items appear, will catch more real conflicts earlier than simpler rules at the same level of clutter. The simpler rules are rendering everything, rendering by eccentricity, rendering by time-to-collision, or rendering by an occupancy threshold. We further claim that in a simulator study it will preserve driver situation awareness in dense traffic, which is where render-everything overlays were shown to harm it.

For each candidate agent $i$ (a tracked road user or an inferred hidden one), we define the score

$$ s_i = P(\text{conflict}_i \mid \text{forecast}) \cdot \big(1 - P(\text{aware}_i)\big), $$

where $P(\text{conflict}_i \mid \text{forecast})$ is the probability, summed over the agent's predicted future modes, that it comes within a safety radius of the ego vehicle's planned path within the forecast horizon. $P(\text{aware}_i)$ is the estimated probability that the driver already knows about the agent. That estimate comes from gaze, if available, or from a visibility-and-eccentricity proxy otherwise, and it is near zero for occluded agents by construction. The display renders the top-$k$ agents whose score exceeds a threshold, with $k$ fixed as a clutter budget.

The new elements are the conflict-and-awareness selection rule, the treatment of occluded agents as first-class candidates, and calibration of the conflict probability. None of the forecasters in the file, nor any of the AR studies, does this.

Relation to Prior Work

Two lines of work motivate the gap. On the display side, Gao et al. showed that highlighting helps peripheral objects in light traffic ($p<.0001$) and hurts central objects in dense traffic ($p=.02$). They called for "selective highlighting" but gave no selection mechanism, and they only highlighted visible, static boxes. The PLOS ONE display study found that top-down surround-view screens drew more fixations but produced lower situation awareness than AR, which argues against moving occluded hazards to a dashboard map. Wang et al. drew future reservation slots on the windshield. Their slots came from a central schedule that assumes full connectivity and no occlusion, so their system has no branching intent and no hidden pedestrians.

On the model side, the method comes from forecasters that already handle occlusion. Scene Informer places query points in blind spots and infers hidden agents from the behavior of visible ones, for example a car stopping at a green light. It forecasts their likely paths on the Waymo Open Motion Dataset. Wei and Lee's switching dynamical model tracks belief "slots" inside line-of-sight shadows and reports calibration. V2VNet and CoBEVT recover occluded objects by fusing compressed features shared between vehicles. All of these systems feed only a robot planner. We reuse them unchanged as the source of candidates and forecasts, and add the layer that decides what a human sees.

Two papers constrain the design. Scene Informer's conservative over-prediction means occupancy cannot be shown directly. RouteFormer shows that driver gaze predicts intent but fails at blind turns. That failure is exactly why the awareness term must treat occluded agents as unnoticed rather than infer awareness from where the driver looks. Conformalized Decision Risk Assessment supplies the calibration idea: it reports that point predictions broke safety guarantees about half the time under bimodal outcomes. We use its conformal recalibration step to make the threshold on $s_i$ meaningful.

Research Program

The work has three parts.

Conflict scoring. We take the per-mode trajectories and mode weights from a forecaster, compute each mode's minimum distance to the ego vehicle's logged future path, and sum the weights of the modes that come within the safety radius. We then recalibrate these probabilities on a held-out split using conformal or isotonic methods, so that a stated 0.3 corresponds to roughly a 30% conflict rate. We report calibration error before and after.

Awareness estimation. In offline datasets without gaze, we use a proxy. An agent counts as "aware" if it has been continuously visible to the ego sensors, within a central eccentricity band, for at least a set duration. Occluded or newly inferred agents are always "unaware." In the simulator study, a gaze tracker replaces the proxy with fixation history. This lets us test whether real gaze adds anything over the proxy.

Rendering primitives for occluded agents. Floating a box over a wall misregisters and invites fixation, so an inferred agent is drawn as a ground-plane emergence contour at the edge of the occluder. The contour is shaped by the agent's predicted modes and scaled by conflict probability. Visible agents get a short path stub along their highest-conflict mode.

Baselines

The naive reference is rendering nothing. It sets the floor for situation awareness and take-over time in the human study and defines zero recall offline.

Render-all draws every tracked and inferred agent with its most likely path. This replicates the condition Gao et al. found harmful, and we expect it to win on recall while losing on clutter.

The eccentricity rule follows Gao et al.'s own positive finding by highlighting only peripheral visible objects. It is the strongest competitor from the human-factors literature because it is a selection rule that was already validated experimentally.

Time-to-collision selection draws an agent when constant-velocity extrapolation predicts contact within a few seconds. It is the strongest engineering competitor because it is cheap, interpretable, and close to what deployed warning systems do.

Occupancy-threshold selection draws inferred hidden agents whose Scene Informer occupancy exceeds a threshold. This is the strongest existing method for the occluded case. The paper reports a 2.37 m reduction in final displacement error under limited observability on the Waymo Open Motion Dataset, and a 10.5% to 27.5% increase in occupancy detection accuracy over baselines. No reported result was found on Lacuna for any alert-selection metric (conflict recall or precision at a clutter budget) for this or any other method.

Every selector receives the same candidates and forecasts from the same backbone, on the same splits, with the same clutter budget $k$. Each selector's single threshold is tuned on the same validation split to hit that budget. Beating a baseline means higher recall of true conflicts, with longer mean warning lead time, at a matched average number of overlays per frame. In the human study it means better situation-awareness scores than render-all in dense traffic and faster take-over than no overlay on occluded-emergence events.

Three ablations isolate the key ingredients. The first replaces the multi-modal conflict probability with the single most likely mode, which tests whether multiple futures matter. The second drops the awareness term. The third skips recalibration.

Why not just…?

Why not just use time-to-collision on a constant-velocity extrapolation? This is the most serious threat, and for visible agents on straight roads it may match our method. It has two structural blind spots. First, an inferred hidden agent has no observed velocity, so time-to-collision is undefined until the agent emerges, which is too late. Second, at bifurcations, extrapolating the current heading misses turns, which is the averaging problem described in Multiple Futures Prediction. We will report results split into straight-road visible conflicts, intersection conflicts, and occluded-emergence conflicts. If time-to-collision ties our method on the first two splits, we will say so and narrow the claim to occlusion.

Why not just follow Gao et al. and highlight peripheral objects only? The eccentricity rule ignores both intent and occlusion. It highlights a peripheral parked car that will never move and cannot highlight a pedestrian behind a truck. Its evidence also comes from light traffic. The dense-traffic comparison in the human study tests directly whether conflict-based selection keeps the benefit where the eccentricity rule has no support.

Why not just threshold Scene Informer's occupancy map? Its authors report that it over-predicts occupancy as visibility drops. A threshold set for good recall in clear scenes would flood the display in occluded ones. Occupancy also says nothing about whether the hidden agent's future path crosses the ego's path. The occupancy-threshold baseline at matched budget, broken down by visibility level, settles this question.

Why not just put hidden hazards on a dashboard bird's-eye map? The PLOS ONE study found that such displays drew more fixations ($P<0.001$) yet produced lower situation awareness and slower take-over. We include a bird's-eye condition fed by the same selector, so that the comparison separates the effect of selection from the effect of placement on the windshield.

Proposed Experiments

Offline evaluation. We evaluate on the Waymo Open Motion Dataset, using line-of-sight occlusion masks in the limited-observability setting of Scene Informer and the dataset paper. Ground-truth conflicts are agents whose logged futures come within the safety radius of the ego's logged path. The metrics are:

  • conflict recall at a matched mean overlay count;
  • false-overlay rate;
  • mean warning lead time before the conflict, or before an occluded agent emerges;
  • calibration error of the conflict probability, with the switching-dynamics model as a reference for forecast calibration (it reports vehicle normalized calibration error 2.67 and average displacement error 3.75 m).

Cooperative arm. On OPV2V we replace inferred hidden agents with detections from cooperative fusion and test whether shared views cut false overlays relative to single-car inference. OPV2V reports attentive intermediate fusion average precision of 0.734 versus 0.406 without fusion. We also add 100 ms of transmission delay, the level V2V4Real found to notably degrade real-world detection, and measure the effect on lead time.

Human study. We replay OPV2V's CARLA scenarios in a driving simulator with a gaze tracker, following Gao et al.'s protocol of object-wise situation-awareness probes, fixation logging, and dense versus light traffic, plus take-over events in which an occluded pedestrian or vehicle emerges. The conditions are no overlay, render-all, eccentricity rule, time-to-collision rule, our selector with the gaze-proxy awareness term, our selector with real gaze, and our selector on a bird's-eye screen.

Decisive checks. There are three. First, at a matched clutter budget, our selector must beat time-to-collision and occupancy thresholding on recall and lead time for occluded-emergence conflicts offline. Second, in dense traffic it must not show the drop in situation awareness that render-all shows. Third, it must shorten take-over time on occluded-emergence events relative to no overlay. If the single-mode ablation matches the full selector, then multiple futures are not earning their cost, and the claim reduces to occlusion-aware selection alone.