A robot moving through a crowd cannot see everything. Other people block its laser scanner, so someone can step out from behind a group with almost no warning. One promising fix treats the visible pedestrians as sensors: if a person ahead suddenly swerves, perhaps they are avoiding someone the robot cannot see. The method that pursues this most directly, Occlusion-Aware Crowd Navigation Using People as Sensors, raises navigation success from 79% to 91% in partially observable simulation, against 97% for a robot that sees everything. That idea carries a flaw. Pedestrians also swerve to avoid the robot itself. A model that reads every swerve as evidence of a hidden person will invent phantom obstacles exactly when the robot is close to people. The robot then brakes or freezes, and its own braking changes how people move around it.
We propose to separate the part of each pedestrian's motion that the robot and the other visible agents already explain from the part they do not, and to infer hidden people only from that unexplained remainder. Put simply, the robot first predicts how a pedestrian would move given everything it can see, including itself. It then treats only the surprise as a clue about what it cannot see. The claim we test is concrete. On a reciprocal crowd benchmark, where humans react to the robot, this explain-away step should cut phantom-obstacle errors near the robot and reduce freezing without losing collision safety. It should also hold up when the robot's own driving policy changes, which is the case where a model that absorbed the robot's influence implicitly should break.
Core Claim
Existing people-as-sensors models learn a map from the observed trajectories of visible agents to the occupancy of hidden space. Examples are the latent-matching variational autoencoder of Mun et al., the discrete-latent driver sensor model of Multi-Agent Variational Occlusion Inference Using People as Sensors, and the inverse-reinforcement-learning inversion in Behavior Planning of Autonomous Cars with Social Perception. The two driving papers work on road networks, where the ego car rarely dominates a neighbour's behaviour. Mun et al. works in crowds, where it often does. In a crowd, pedestrian motion depends on the hidden state and also on the robot's own trajectory. The robot's trajectory comes from the robot's policy, and the policy changes during reinforcement learning and at deployment. A sensor model trained on data from one robot policy therefore mixes up "a hidden person is there" with "this is how people react to this particular robot." We call this reciprocal confounding. The research file flags it as unaddressed, and we found no paper that addresses it.
The fix is an explicit forward model of pedestrian response. Let $v_i^t$ be pedestrian $i$'s observed velocity at time $t$. Let $\hat v_i^t = f(h_i, N_i, s_R)$ be the velocity predicted by a response model $f$, given the pedestrian's own history $h_i$, their visible neighbours $N_i$, and the robot's state and recent motion $s_R$. The occlusion model then receives the residual
$$r_i^t = v_i^t - f(h_i, N_i, s_R),$$
together with the occlusion geometry. It does not receive raw trajectories. When the residual is large, the pedestrian is doing something the visible scene cannot account for, and that is the evidence a hidden agent is worth considering. When the residual is small, the pedestrian's swerve has been explained away, and the swerve says nothing about occluded space.
Three pieces are prior work. The response model $f$ belongs to the interaction-graph family (Relational Graph Learning for Crowd Navigation already learns human motion predictions for lookahead planning). The occlusion decoder reuses Mun et al.'s cross-modal supervision from a god-view teacher. The idea that collision avoidance is an informative prior during estimation comes from GTP-SLAM, although that work handles sensor noise rather than fully hidden agents. What is new is where the robot's influence sits. It is modelled explicitly and subtracted before occlusion inference, rather than absorbed implicitly or ignored.
Research Program
We build on the CrowdNav simulator, with humans driven by ORCA, the standard analytical collision-avoidance rule in which each agent picks a velocity that avoids others assuming they do the same. We add line-of-sight ray casting in the style of Mun et al., so the robot sees only unoccluded pedestrians. CrowdNav's switch between a visible and an invisible robot gives a natural control. With the robot invisible, humans ignore it and there is no reciprocal confounding. With it visible, confounding is present. Any method that is actually unconfounded should lose little when we move from invisible to visible. A confounded method should lose more.
The response model $f$ is trained on god-view simulation data, which includes the robot. The pedestrian's neighbourhood is restricted to agents the robot can see, so that $f$ predicts from the same information available at test time. The occlusion model is trained with Mun et al.'s cross-modal latent matching, taking residuals as input. It outputs occupancy of occluded cells, and, following the argument of BiVO that planners need time-stamped paths, a short set of candidate trajectories for hidden pedestrians. We deliberately keep the planner the same as Mun et al.'s PPO policy (proximal policy optimization, a standard on-policy reinforcement learning algorithm). Any gain then comes from the sensor model, not from planning.
Robot-policy shift is the stress test. Sensor models are trained on rollouts from one robot policy and evaluated on rollouts from another. We use two contrasts: an early-training PPO policy against the converged one, and a deliberately assertive policy against a deliberately timid one. Under a shift, a model that has memorised how people react to the training robot should mispredict. The residual model conditions on the robot's actual motion through $f$, so it should degrade less. We also add static occluders such as pillars and L-shaped corners, which Mun et al. lists as future work. These let us check that the residual evidence combines with fixed sightlines, but they are a secondary setting.
Baselines
The naive reference point is the limited-sensor planner that acts only on visible pedestrians and does no occlusion inference. Mun et al. report 79% success for it in their partially observable simulation. We rerun it in our environment rather than quote the number, because our scenes differ. The oracle upper bound is the same planner given god-view input; Mun et al. report 97%.
The strongest existing method is PaS-VAE from Occlusion-Aware Crowd Navigation Using People as Sensors, which reports 91% success and 13.91 s navigation time. We retrain it from the released design on exactly our training rollouts, with the same planner, the same number of PPO environment steps, and the same hyperparameter search budget as our method, over the same number of seeds. It is the right competitor because it is the only method in the file that covers all three directions, and it is precisely the model we claim is confounded. Beating it means two things. First, a statistically significant reduction in phantom-occupancy rate and freezing rate in the visible-robot setting, at equal or lower collision rate. Second, a smaller drop in success under robot-policy shift. Matching its success rate alone does not count.
The critical ablation is PaS-VAE with the robot's state and recent motion added as extra input, and no residual. It sees the same information as our model but must learn the explaining-away implicitly. If it matches the residual model under policy shift, the explicit subtraction is unnecessary, and we will report that. We also run a hand-written heuristic that discards evidence from any pedestrian within a fixed radius of the robot or on a converging course, with the radius tuned on validation. Finally, we adapt the worst-case risk-particle approach of Occlusion-Aware Risk Assessment for Autonomous Driving in Urban Environments to free space: hypothetical pedestrians at maximum walking speed fill every shadow. That paper reports a 95th-percentile collision rate falling from 12.64% to 2.61% in driving, but at the cost of heavy braking at every blind corner.
For context we note that MP-RGL reports 96% success with five humans in the fully observable invisible-robot setting. It solves a different problem and is not a direct competitor.
Why not just…?
Why not just give PaS-VAE the robot's state as an input? This is the real threat. With enough varied training data, a conditioned network could learn the same subtraction. Our argument is about distribution shift. The conditioned network only learns how people react to the robot trajectories it saw during training. The residual model factors that reaction into $f$, which is trained on god-view data and can be trained on rollouts from many policies without hidden-state labels. The policy-shift experiment settles the question. If the conditioned baseline degrades as little as ours, we concede that the structure buys nothing beyond sample efficiency and measure that sample efficiency directly.
Why not just ignore pedestrians near the robot? Discarding nearby pedestrians removes the confounded evidence, but it also throws away the most informative observers. A person walking toward a blind corner next to the robot is often the one who can see around it. We expect the tuned gating heuristic to lower phantom obstacles and raise missed hidden agents. Stratifying occupancy recall by pedestrian–robot distance will show whether that trade actually occurs.
Why not just treat every shadow as occupied? Worst-case reasoning is safe, but in crowds shadows are everywhere. BiVO and Mun et al. both describe it producing freezing, and Yu et al.'s virtual stop signs illustrate the conservatism. We include it to anchor the safety end of the trade-off. We expect it to show the lowest collision rate and the highest timeout rate. Our method should dominate it on time-to-goal at comparable collision rates, or we report that it does not.
Proposed Experiments
The main evaluation crosses three factors: robot visible or invisible, training and test robot policies matched or shifted, and human density at 5, 10 and 20 people. Static occluders are added in a secondary block. We report success, collision, and timeout (freezing) rates, navigation time, and discomfort frequency, meaning intrusions into personal space as used in the LM-SARL evaluation.
Two perception metrics come from ground truth. The first is phantom-occupancy rate: the probability mass assigned to empty occluded cells, split by whether a visible pedestrian within 2 m of the robot was deviating at that moment. The second is recall of hidden pedestrians who later enter the robot's path.
The decisive checks are three. First, in the visible-robot setting, PaS-VAE's phantom-occupancy rate should rise specifically in the robot-proximate stratum, and ours should not. This directly tests whether the confound exists at all. If PaS-VAE shows no such rise, the premise fails and we stop. Second, the gap between PaS-VAE and the residual model should widen under robot-policy shift. Third, the residual model should close part of the gap to the 97% oracle without raising collisions above PaS-VAE's.
A small Turtlebot study replays Mun et al.'s corner scenario with a confederate walker who deliberately yields to the robot. This is a qualitative check that the robot does not brake for a phantom. We make no statistical claims from it.
References
- Occlusion-Aware Crowd Navigation Using People as Sensors
- Multi-Agent Variational Occlusion Inference Using People as Sensors
- Behavior Planning of Autonomous Cars with Social Perception
- Relational Graph Learning for Crowd Navigation
- GTP-SLAM: Game-Theoretic Priors for Simultaneous Localization and Mapping in Multi-Agent Scenarios
- Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models
- Occlusion-Aware Risk Assessment for Autonomous Driving in Urban Environments