Provenance-Aware Scene Graphs for Cooperative Motion Forecasting Under Failures

When several connected vehicles share what they see in order to predict where everyone nearby will move, a car can vanish from the shared picture for two very different reasons. It may have stopped being observed: a peer's radio dropped, its sensor failed, or the car slipped behind a truck. Or it may still be observed perfectly well and simply have stopped moving, for example a stalled vehicle that should now be treated as part of the road. Current motion forecasters cannot tell these cases apart. They read a missing input as "no agent here" or as "agent still moving as before." The first error deletes a hazard. The second invents motion that is not happening. Swarm-robotics work has long handled dead units by folding them into the environment as static obstacles. Neural forecasters have no such mechanism, because they never record who could have seen what.

We propose that each vehicle exchange a compact object-level scene graph rather than dense image-like feature maps. In this graph, nodes are tracked road users. Explicit provenance edges record which peer observed each node, with what confidence, and which regions each peer was actually able to see. A forecaster reading this graph can then infer, for each road user, whether it is observed and moving, unobserved and extrapolated with growing uncertainty, or observed and stopped, in which case it is absorbed into the static scene. The core claim is narrow and testable. When peers fail, the information that separates "absence of evidence" from "evidence of absence" is sensor coverage, not model capacity. A graph that carries coverage explicitly should beat equally trained forecasters that see only the surviving observations, and the advantage should grow with the fraction of failed peers.

Where this sits in prior work

Three strands supply the pieces, and none of them combines them. Structured interaction forecasters represent the scene as a typed graph. HDGT builds a heterogeneous graph of agents, lanes and signs, uses relative-pose edges, and runs relation-specific attention. GraphAD rewires agent-to-agent edges using predicted-trajectory proximity. LookOut uses a graph network scene encoder to drive diverse futures and contingency plans. All three run centrally on one vehicle and assume every agent is continuously tracked. We reuse HDGT's typed-edge design as our backbone, and we contribute what happens when the inputs come from unreliable peers.

Cooperative perception handles unreliable peers, but only for detection. Where2comm has agents send features from high-confidence regions and "request maps" for their blind spots, which is exactly the coverage signal we want. However, it uses that signal only to decide what to transmit, over dense bird's-eye-view grids. It never feeds coverage into a forecaster. OPV2V and V2X-Sim provide the simulated multi-vehicle benchmarks and fusion baselines, again for detection, tracking and segmentation rather than forecasting. V2X-Sim's negative finding constrains our design: attention-based collaboration (When2com, Who2com) underperformed because attention favors correlated views over complementary ones. Our provenance edges are typed by observer, so a lone complementary observation cannot be washed out by redundant ones. This is a hypothesis we test rather than assume.

The fault-tolerance literature supplies the remaining two ideas. The first is absorbing failed units into the environment, from the Guaranteed Voronoi coverage work of Papatheodorou et al. and the passive-adaptation results of Yu et al., both summarized in the file without a linkable entry. The second is the fault-tolerance ratio of Symbolic Synthesis of Fault-Tolerance Ratios, defined as the smallest fraction of failed components that breaks a global specification. Its symbolic synthesis does not extend to neural predictors, as that paper's own scope makes clear. We therefore borrow only the quantity and measure it empirically. Digital self-healing systems such as CARD and MorphAgent show that rewiring around failed nodes preserves performance in LLM pipelines, with MorphAgent holding 40–54% under 80% agent failure. They motivate the effort, but neither has spatial coverage, so neither can face our core ambiguity.

Core Claim

Each road-user node in the shared graph receives a latent status with three values: active (observed and moving), occluded (not currently observed by anyone whose coverage includes it), or absorbed (observed and stationary beyond a dwell threshold, re-typed as a static obstacle node connected to lanes the way HDGT connects signs). Status is inferred from provenance edges. These carry per-peer observation confidence, per-peer coverage polygons summarized as region-to-node "could-see" edges, and peer liveness, meaning time since the last message.

Occluded nodes keep their dynamic edges, but their predicted distributions are widened with time since last observation. Absorbed nodes drop their own forecast heads and lose outgoing social edges, and other agents route around them. The claim is that this status inference, driven by explicit coverage, is what makes forecasting degrade gracefully under peer failure. A forecaster given the same surviving tracks without coverage cannot learn the distinction, because the two situations produce identical inputs.

Research Program

We build a decentralized forecasting stack in which every connected vehicle maintains its own provenance-aware graph from its own sensing plus whatever peer messages arrive. Each message is a list of tracked objects with states and confidences, plus a coarse coverage polygon. That is orders of magnitude smaller than feature maps, and its size can be logged against Where2comm's 100,000× communication reduction for context. The forecaster is an HDGT-style heterogeneous graph transformer extended with provenance edge types and a three-way status head. Status is trained with labels available in simulation: ground-truth visibility per peer, and ground-truth velocity for stalls.

Faults are injected along three axes, applied independently and jointly. The first is peer communication loss, as a fraction of connected vehicles going silent. The second is sensor failure, where a peer keeps transmitting but its coverage shrinks or its confidences become noisy. The third is physical stalls, where a vehicle decelerates and stops in lane.

The analysis centers on an empirical fault-tolerance curve. For each method we sweep the failed-peer fraction from 0 to 0.8. The empirical fault-tolerance ratio is the smallest fraction at which miss rate on safety-relevant agents exceeds a fixed threshold, set at twice the full-information miss rate. This mirrors the definition in Kouvaros et al. without claiming formal guarantees.

Proposed Experiments

The primary testbed is V2X-Sim, because it couples SUMO traffic with CARLA sensing across multiple vehicles and roadside units and already ships tracking ground truth. Using the same CARLA and SUMO pipeline, we generate additional episodes with scripted stalls, since stalls are rare in existing logs. Forecasting targets are 3-second futures from 1-second histories, extracted from the ground-truth tracks.

OPV2V scenes serve as a second simulated domain to check transfer across towns, including the Culver City replica. As a semi-real check, we use the drone-captured INTERACTION dataset. There, we designate a subset of vehicles as connected and synthesize each one's coverage by ray-casting occlusion from the top-down boxes. This gives real interactive behavior, with its 13,375 tight-conflict pairs, under controlled fault injection.

Metrics are minADE and minFDE over six modes, and miss rate restricted to agents within 30 m of the ego vehicle's path. Two failure-specific measures are added. Ghost-motion error is the predicted displacement assigned to stalled vehicles. Dropped-hazard rate is the fraction of truly present agents within that 30 m path corridor that receive no forecast. We also report status classification accuracy and bytes transmitted per frame.

For closed-loop relevance, a simple sampling-based planner consumes each forecaster's output in CARLA. We count collisions and progress per collision, the metric LookOut used, where it reported a 21% gain over its baselines on its own ATG4D and LidarSim setup. That number is not comparable to ours.

Baselines

The naive reference is single-vehicle HDGT with no communication: each car forecasts from its own sensors. It sets the floor that cooperation must beat. V2X-Sim and OPV2V both show large single-agent gaps in detection; for example, OPV2V reports [email protected] rising from 0.406 to 0.734 with attentive intermediate fusion. We have no reported forecasting number for this setting.

The upper reference is HDGT given all ground-truth tracks. On the original INTERACTION benchmark, HDGT reports minADE 0.1676 against a prior best of 0.2130. We will rerun it on our splits rather than quote it, because our horizons and agent subsets differ.

The strongest competitor is late track fusion plus robust training. This is the union of all surviving peers' tracks fed into the same HDGT backbone, trained with exactly our fault-injection distribution, so it sees random node deletion, peer silence and stalls during training. It is strongest because it is what a practitioner would do first and needs no new representation. It receives the same architecture size, data, injected-fault schedule and hyperparameter search budget, which is equal trials on the same validation split.

The second strong competitor is feature-level cooperation: Where2comm fusion producing a bird's-eye-view map, followed by a grid-based forecaster head. It is trained on the same episodes and compared at matched and at unconstrained bandwidth. We found no reported result on Lacuna for Where2comm or any cooperative-perception method on trajectory forecasting under peer dropout, so all comparisons are newly run.

The key ablation strips provenance edges from our model while keeping the status head and training identical. This isolates whether coverage information, rather than the extra head or graph typing, causes any gain.

Beating the strongest baseline means three things. First, a higher empirical fault-tolerance ratio. Second, lower miss rate at failure fractions of 0.4 and above, with confidence intervals over five seeds. Third, a ghost-motion error near zero on stalls, while staying within noise of it at zero failures.

Why not just…?

Why not just train the existing forecaster with random agent dropout? This is the real threat, and it is the baseline we tune hardest. Our argument is informational rather than empirical so far. For a single agent, "unobserved because its observer failed" and "no longer present" produce identical surviving inputs, and "stalled" and "occluded" become identical once the track ends. No amount of augmentation can separate cases that look the same at input. The provenance ablation settles it. If the augmented baseline matches our full model, or the provenance-free ablation matches it, the claim is wrong and coverage is not the missing ingredient. The likely partial outcome is that augmentation closes most of the gap at low failure rates. We expect our advantage to concentrate in stalls and high failure fractions, and we will report it that way.

Why not just coast missing tracks with a constant-velocity or Kalman extrapolation? Coasting is reasonable for short gaps, and it is the default for occluded nodes inside our own model. It is wrong for stalls by construction, since it keeps a stopped car moving, and its error grows without bound over long silences. It also cannot remove a node that has truly left the scene. We include a coasting variant of the late-fusion baseline and report ghost-motion and dropped-hazard rates separately, so any regime where coasting is enough is visible.

Why not just run Where2comm-style feature fusion and put a forecaster on top? Feature fusion keeps more raw information, and at generous bandwidth it may win on clean scenes. But it represents absence implicitly in a grid. V2X-Sim found that attention-based fusion favors correlated views over complementary ones, which is precisely the failure we worry about when one surviving peer holds the only view of a hazard. Matched-bandwidth comparisons and per-agent analysis of sole-observer cases test this directly. If feature fusion wins even when bandwidth is matched, the object-level graph is the wrong carrier, and we would add coverage maps to feature fusion instead.

Decisive Checks

Three results would confirm the idea. The full model should beat the dropout-augmented late-fusion baseline on miss rate at failure fractions of 0.4 and above in V2X-Sim, and also on the INTERACTION synthetic-coverage variant. The provenance-free ablation should lose most of that gain. On stalls, our planner should collide less, and progress should not fall, relative to every baseline in closed-loop CARLA.

Any of three outcomes would falsify it. The augmented baseline matching us across the whole failure sweep would do so. So would the ablation matching the full model. So would gains appearing only in simulation with perfect coverage polygons. To guard against the last case, we rerun all comparisons with coverage polygons corrupted by realistic occlusion-estimation noise.