A self-driving car at a blind intersection cannot see what is behind the parked truck, but the drivers who can see it often reveal it. When a nearby car brakes hard for no visible reason, something is probably there. Prior work has shown that treating surrounding drivers as proxy sensors can replace the extreme caution of "assume a hazard hides in every blind spot" (Behavior Planning of Autonomous Cars with Social Perception). The same work has a known failure. If a witness driver is distracted and does not brake, the ego car concludes the hidden crossing is clear. That inference is asymmetric: a reaction is informative, but the absence of a reaction is only as informative as the witness is reliable. Simulators make the problem worse. Scripted traffic agents always react, so any policy trained against them learns to trust silence completely. HABIT shows how badly scripted-agent training transfers: leaderboard agents with zero collisions on standard CARLA reach 5.24 to 7.43 collisions/km once pedestrians move like real humans.
We propose to treat witness reliability as a sim-to-real quantity that must be estimated, not a constant the simulator happens to hard-code. The model is a spatio-temporal graph, meaning a network whose nodes are agents and whose edges are their changing pairwise relations over time. Its nodes include the visible road users and the hidden map regions. Its edges connect each witness to the occluded zones inside that witness's line of sight. The graph predicts which hidden zones are occupied. A structural rule lets a visible reaction raise the estimated hazard freely, while the absence of a reaction can lower it only in proportion to how often witnesses actually react. That reaction rate is calibrated against real traffic data, not against the simulator. The occupancy belief feeds a high-level policy that chooses among discrete driving intents such as "creep and probe", "yield", or "proceed". A low-level controller tracks each intent under randomized actuator delay. The main claim is testable: this asymmetric, reliability-calibrated use of social evidence keeps most of the speed advantage that active occlusion reasoning has over reachability-based caution, and it does not collapse when witnesses stay silent.
Core Claim
Social occlusion inference fails in the real world mainly because it treats witness silence as strong evidence. Simulators hide this flaw because their agents react perfectly. A planner that bounds the evidential weight of silence by an estimated reliability $\hat\rho$ should behave safely under inattentive witnesses and realistic humans, while staying far less conservative than worst-case reachability.
For each occluded zone $z$, the planner uses
$$p_{\text{used}}(z) = \max\big(\hat p_{\text{graph}}(z),\ (1-\hat\rho_z)\,p_{\text{prior}}(z)\big).$$
Here $\hat p_{\text{graph}}(z)$ is the graph network's predicted probability that zone $z$ holds a road user, given witness behavior. The term $p_{\text{prior}}(z)$ is a map-based prior occupancy, such as a particle or reachability estimate. The quantity $\hat\rho_z$ is the estimated probability that the witnesses with line of sight to $z$ would visibly react if it were occupied. When no witness covers $z$, $\hat\rho_z = 0$ and the rule falls back to the prior. The graph can raise the hazard above the prior without limit, but it can lower it only down to the reliability-discounted floor.
Relation to Prior Work
Four groups of prior work play different roles here.
The social-perception planner of Behavior Planning of Autonomous Cars with Social Perception motivates the proposal. It infers hidden pedestrians from one human driver's braking, using inverse reinforcement learning (recovering a driver's goals from their observed actions) and probabilistic model predictive control. It is tested only in a toy simulation, and the dossier identifies its collapse under non-optimal witnesses. We replace the single-witness optimal-driver model with a learned graph over many witnesses, and we make silence explicitly unreliable.
CMetric and StylePredict show that disruptive or cautious behavior shows up as changes in graph centrality. That supports the premise that relational structure carries hazard signal. Inferring Latent Temporal Sparse Coordination Graph motivates inferring edges when geometry alone is insufficient. We avoid full latent-edge inference: witness-to-zone edges are computed geometrically from map and pose, which keeps the graph small and interpretable.
Two occlusion planners constrain the design. The particle risk field of Occlusion-Aware Risk Assessment seeds hidden lanes with phantom agents. The game-theoretic planner of Safe Occlusion-Aware Autonomous Driving via Game-Theoretic Active Perception treats hidden agents as adversaries. Both are safe but conservative. We reuse the first as $p_{\text{prior}}$, and we do not claim to beat either on worst-case guarantees.
Pad-AI supplies the control split. Its reinforcement-learning policy picks semantic intents that polynomial planners turn into feasible trajectories, over polyline scene encodings. Its authors defer sim-to-real transfer, and it uses no witness reasoning. We keep its interface and change what feeds it.
Cooperative perception systems such as V2VNet and Where2comm remove occlusion by sharing sensor features between vehicles. We use them only as data sources for ground truth, because the target setting is traffic with no connected vehicles. NaviSTAR is the closest graph-plus-real-deployment work. It is a monolithic policy that assumes detectable agents, and its authors observed that real people do not follow the scripted crowd model (ORCA) used in its simulation. That finding motivates us.
Research Program
Graph. At each timestep the graph holds three kinds of nodes: visible vehicles and pedestrians, the ego car, and occluded zones. Occluded zones are map lane segments outside ego visibility. Witness-to-zone edges exist when a witness's estimated field of view covers the zone. Agent-to-agent edges follow proximity, as in GraphRQI. A small spatial attention layer, followed by a recurrent unit over roughly two seconds of history, produces zone occupancy $\hat p_{\text{graph}}$ and a predicted motion for any occupied zone. We choose a small attention-plus-recurrence model over a dense graph transformer because of the onboard latency concern raised for NaviSTAR.
Reliability. The reliability $\hat\rho$ is a function of witness type (driver or pedestrian), distance, and viewing angle. It is fit, where possible, on real traffic from DAIR-V2X, a real-world dataset in which roadside units observe road users hidden from the vehicle. Infrastructure detections give ground truth for zones occluded to the vehicle, so we can measure how often visible vehicles decelerate when such a zone is occupied. If DAIR-V2X lacks enough sequential coverage for this, we report that and fall back to a HABIT-derived estimate. In simulation, training uses CARLA scenes with a randomized fraction of inattentive witnesses, meaning scripted drivers that do not react. The gap between the reliability learned in simulation and the reliability measured on real data is itself a reported result.
Control. A high-level policy picks one intent every 100–200 ms from the Pad-AI set, given $p_{\text{used}}$ and the visible graph embedding. A low-level trajectory tracker runs at high rate. During training, actuation delay and tire friction are domain-randomized, meaning they are sampled from ranges instead of fixed. A probing intent is admissible only if, under the sampled delay, the ego car can still stop before the conflict zone should a hidden agent appear. This check targets the danger, noted in the dossier's gap analysis, that probing maneuvers become unsafe under real latency.
Baselines
The naive reference is a creep-and-stop rule that crawls at a fixed low speed whenever any conflict zone is occluded. It bounds the safety-versus-speed trade from the cautious side.
The strongest safety-oriented baseline is the particle risk-field planner of Occlusion-Aware Risk Assessment. It reports a 95th-percentile collision rate of 2.61% and a median of 1.45% across 73 OSM intersections, though that is on its own layouts, not ours. We rerun it in our CARLA scenes with its 1.5-second propagation horizon.
The strongest efficiency-oriented baseline is Pad-AI. It reports a 100% success rate in its CARLA occlusion scenarios and 37–61% higher average speed than reachability methods. We found no reported result on Lacuna for Pad-AI under HABIT pedestrians or inattentive witnesses.
For realistic pedestrians, the reference point is InterFuser on HABIT: 5.24 collisions/km, 10.96% injury risk, and a false-positive braking rate of 0.33.
The trusting-witness ablation sets $\hat\rho = 1$, which turns the proposal into a learned version of classic social perception.
The flat-encoder ablation feeds neighbor state histories into Pad-AI's polyline encoder in place of the graph. It tests whether the graph structure matters, as opposed to simply having witness information.
All learned methods share the same scenes, the same randomization of witness attentiveness and actuator delay, the same reinforcement-learning sample budget, and the same hyperparameter search budget. Beating the field means two things together. First, strictly fewer collisions per km than Pad-AI and the trusting ablation under inattentive witnesses and HABIT pedestrians. Second, average speed within 15% of Pad-AI's in reliable-witness scenes, and above the particle planner's.
Why not just…?
Why not just use the particle risk field and tune it? It never trusts silence, so it cannot be fooled by a distracted witness. That makes it the strongest competitor on safety. Its cost is speed, and Pad-AI's reported 37–61% speed gap over reachability methods quantifies how large that cost can be. Our rule contains this planner as the $\hat\rho = 0$ special case. If the fitted $\hat\rho$ turns out near zero on real data, the honest conclusion is that social evidence buys little in real traffic, and the experiments are designed to report that outcome.
Why not just add neighbor histories to Pad-AI? This is a real threat. A flat encoder may learn witness cues just as well at the fixed agent counts of our scenes. The flat-encoder ablation settles it. We expect the graph to help mainly when the number of witnesses at test time differs from training, the regime where InforMARL shows graph policies generalize. We will report both matched and mismatched agent counts. If the graph advantage appears only in the mismatched case, we will say so.
Why not just randomize witness attentiveness and let an end-to-end policy calibrate itself? A policy trained with randomized attentiveness might learn the discount implicitly. The randomized-trusting ablation tests this: it uses the same randomization with no max-floor rule. The explicit rule has two advantages beyond performance. It allows $\hat\rho$ to be refit from real data without retraining the policy. It also gives a guarantee that is checkable by inspection, since silence can never push $p_{\text{used}}$ below the discounted prior. HABIT's evidence that randomization over simulator behavior fails under realistic humans suggests that implicit calibration will track the simulator's attentiveness distribution, not reality's. That is a hypothesis, and this ablation is where it is tested.
Proposed Experiments
Silent-witness test (decisive). In CARLA blind crossroads and T-intersections, we build matched scene pairs. In one scene of each pair, a hidden pedestrian or vehicle is present and a visible driver reacts to it. In the other, the hazard is present and the driver does not react. The trusting ablation should collide in the silent case. The proposal should not, and should still pass faster than the particle planner in the reactive case and in empty scenes. Metrics are collisions/km, time-to-cross, false-positive braking rate, and, following HABIT, injury risk.
Behavioral reality check. We replace scripted pedestrians with HABIT motion sequences and compare against the InterFuser reference above.
Reliability transfer. We compare $\hat\rho$ fit in simulation against $\hat\rho$ fit on DAIR-V2X. We then report how much each choice changes safety when the model is evaluated with HABIT pedestrians and inattentive-witness drivers.
Latency stress. We sweep actuation delay beyond its training range and measure how often probing intents lead to stops inside the conflict zone. This is compared with an unconstrained probing policy.
Real hardware. Physical-vehicle validation is outside this proposal's budget. The real-world component is limited to calibrating reliability from real traffic data, and we state that limitation plainly.
References
- Behavior Planning of Autonomous Cars with Social Perception
- HABIT: Human Action Benchmark for Interactive Traffic in CARLA
- CMetric: A Driving Behavior Measure Using Centrality Functions
- Using Graph-Theoretic Machine Learning to Predict Human Driver Behavior
- Inferring Latent Temporal Sparse Coordination Graph for Multiagent Reinforcement Learning
- Occlusion-Aware Risk Assessment for Autonomous Driving in Urban Environments
- Safe Occlusion-Aware Autonomous Driving via Game-Theoretic Active Perception
- Learning Occlusion-aware Decision-making from Agent Interaction via Active Perception
- V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
- Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps
- NaviSTAR: Socially Aware Robot Navigation with Hybrid Spatio-Temporal Graph Transformer and Preference Learning
- GraphRQI: Classifying Driver Behaviors Using Graph Spectrums
- Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation