Cross-View Gaze Cones for Occluded Pedestrians

Whether a pedestrian at the curb has looked toward an approaching car is one of the strongest cues that they are about to cross, or about to wait. Driving systems mostly read that cue from head orientation in a single onboard camera. That fails in exactly the moments that matter: a bus hides the pedestrian's face, it is night, or the face is a few pixels wide. Yet in many of these cases some other sensor can still see the head. A roadside camera mounted on a pole may have a clear view of the face. A LiDAR (a laser scanner that returns 3D points) still sees the shape of the head and shoulders in the dark. This note proposes a small, testable step. Each sensor reports its estimate of where the head points as a calibrated directional distribution, which we call a gaze cone: a most-likely direction plus a learned confidence that controls how wide the cone is. These cones are anchored at a metric 3D head position measured by LiDAR and combined in a shared world frame. The output is the probability that the pedestrian is looking at the ego vehicle.

The core bet is that the hard part is neither the network architecture nor the communication bandwidth. It is knowing how much to trust each viewpoint's head-pose estimate, and putting that estimate in the right place in 3D. If the cones are honestly calibrated, combining them is nearly free, and the combination can be sent between vehicles and infrastructure in a few bytes. If the cones are not calibrated, no fusion scheme rescues them. The proposal is built so that experiments can tell these two situations apart.

Core Claim

A head-pose estimate becomes useful across sensors and viewpoints once it is represented as a direction with a predicted confidence, placed at a LiDAR-measured head location, and combined with a body-orientation prior that is available even when no camera sees the face. We claim this representation recovers the "looked-at-ego" signal under face occlusion and darkness better than three simpler options: the onboard camera alone, picking the single best view, and LiDAR body heading alone. We further claim that the gain shows up downstream in crossing-intention prediction.

Concretely, view $i$ produces a unit direction $\mu_i$ in its own sensor frame and a confidence $\kappa_i \ge 0$. These are the mean direction and concentration of a von Mises–Fisher distribution, which is the spherical analogue of a Gaussian. LiDAR provides a torso-heading prior $(\mu_0, \kappa_0)$. With $R_i$ the known rotation from sensor $i$ to the world frame, the product of these distributions is again von Mises–Fisher, with

$$\kappa\,\mu \;=\; \kappa_0\,\mu_0 \;+\; \sum_i \kappa_i\, R_i\,\mu_i ,$$

so the fused cone is a confidence-weighted vector sum. The quantity we predict is $p_{\text{ego}} = P(\angle(d, h \to e) < \theta)$. Here $d$ is a direction drawn from the fused cone, $h$ is the LiDAR head position, $e$ is the ego vehicle's position, and $\theta$ is a small attention half-angle. The sum shows why calibration carries all the weight. An overconfident wrong view dominates the fused direction, so $\kappa_i$ must actually shrink when the face is dark, small, or turned away.

Prior Work and Where This Sits

Head orientation as an attention proxy is well motivated. Eyes tend to stay near the center of the eye socket while the head turns toward what is being attended (Saliency in VR). Eye, head, and body move in a predictable coordinated order, which supports both gaze estimation at a distance (Dynamic 3D Gaze from Afar) and gaze estimation from upper-body skeletons without detailed facial imagery (Upper-Body Pose-based Gaze Estimation). The cone representation is borrowed from work that projects angular uncertainty cones from facial and neck keypoints (Understanding Compositional Structures in Art Historical Images). That work fixes the cone width by hand. We learn it per observation.

The closest application work is FHPE-Net. It feeds monocular head yaw, pitch, and roll into a recurrent intention model on PIE, and it reports that removing head pose costs 8% accuracy and 10% F1. That result motivates the proposal: head pose matters. Its stated limits also constrain the proposal: it is monocular, it fails in low light, and it has no other viewpoints. The authors themselves suggest adding LiDAR or radar. The Occlusion-Aware Diffusion Model fills occluded frames by generating them from visible bounding boxes. It uses no head pose, and its authors note that it invents implausible paths when behavior changes during the occlusion. That is exactly the case in which a second, real observation of the head is needed.

On the occlusion side, cooperative perception shares information between agents. V2VNet and CoBEVT share intermediate feature maps. DAIR-V2X benchmarks early, intermediate, and late fusion between vehicles and roadside units, and corrects late fusion for time delays. All three share only boxes, occupancy, or map semantics; none of them transmits head orientation. SiMO shows that fused camera–LiDAR models collapse when one modality drops out, and recovers single-modality operation through staged training and random modality dropout. We reuse its dropout lesson directly: the fusion must degrade gracefully when the face-seeing camera disappears. Behavior Planning with Social Perception infers hidden pedestrians from other drivers' braking. That approach is complementary but relies on those drivers being attentive. Our signal comes from the pedestrian directly.

Research Program

The program has three components.

The first is a head-pose estimator that predicts its own uncertainty. It is a standard head-pose regressor with an added output for $\kappa$, trained with the von Mises–Fisher negative log-likelihood. During training it sees augmentations that mimic the target failures: darkening, blur, downscaling to a few-pixel face, and partial masking. A post-hoc temperature on $\kappa$ is fitted on held-out data so that the empirical coverage of a cone matches its nominal coverage.

The second is a LiDAR module. It takes the points inside a pedestrian's 3D box, extracts the head position as the top cluster of points, and regresses torso heading with its own confidence. This gives the cone a metric anchor without monocular depth guessing, and it supplies the fallback prior $\kappa_0\mu_0$ when no camera sees the face. This prior is the physical version of the body–head coupling exploited in Dynamic 3D Gaze from Afar.

The third is the fusion and transmission step. Each agent sends, per tracked pedestrian, its $\kappa_i R_i \mu_i$ vector (three floats) along with the box it already sends under late fusion. The receiver sums these vectors and computes $p_{\text{ego}}$. For delayed messages, the head position is propagated forward as in DAIR-V2X's time-compensated late fusion, and $\kappa_i$ is reduced in proportion to message age. That aging rule is the only free design choice, and it is ablated.

For the downstream test, $p_{\text{ego}}$ and the fused direction replace FHPE-Net's raw yaw/pitch/roll input stream. Everything else in that model stays the same.

Baselines

The naive reference is FHPE-Net's own architecture with the head-pose stream removed. The file reports what this costs on PIE (the 8% accuracy and 10% F1 drop), and this sets the floor that head information must beat.

The strongest existing method for the downstream task is FHPE-Net itself, which reports 90% accuracy and 0.82 F1 on PIE (FHPE-Net). It will be retrained by us on the identical split with the same tuning budget. The only change in our variant is the head input.

For the cooperative, occluded setting, no reported result exists on Lacuna for head-pose or gaze estimation in vehicle–infrastructure cooperative data, so we construct the competitor. It is a best-view late-fusion baseline: each agent runs the same head-pose network, sends yaw and pitch, and the receiver keeps the estimate from the view with the largest face bounding box. This is the obvious engineering answer, and it shares our backbone, compute, and bandwidth class.

A second constructed competitor is LiDAR-only torso heading, which tests whether faces are needed at all.

Two ablations isolate the proposal's key ingredients. The fixed-confidence ablation sets every $\kappa_i$ to a constant, as in hand-set cones, and tests whether learned calibration is what matters. The monocular-anchor ablation places the cone using camera-estimated depth instead of LiDAR head position, and tests whether metric anchoring matters for the angle to the ego vehicle.

We count the method as beating the competition if it improves the angular error of $p_{\text{ego}}$ and its area under the curve over best-view selection on the face-occluded subset, with non-overlapping bootstrap intervals. Downstream, it must reduce the intention-F1 drop under night degradation relative to FHPE-Net, while not falling below FHPE-Net's 0.82 F1 on clean PIE.

Why not just…?

Why not just send yaw and pitch from whichever camera sees the face best? This is the real threat, and it is exactly our best-view baseline. If the camera with the largest face is usually also the most accurate one, best-view selection captures most of the benefit, and our contribution shrinks to calibration plus metric anchoring. We expect it to fail in two situations: when two views are each partially informative (a profile view and a dark frontal view), and when the largest face is the most misleading one, for example a large but motion-blurred face. The per-condition breakdown in the experiments will show which regime dominates. We will report the result even if best-view selection wins on average.

Why not just estimate body heading from LiDAR and skip faces? LiDAR works in the dark and needs no face. However, the "looked before crossing" behavior is precisely a head turn without a body turn. FHPE-Net's false positives come from pedestrians who turn their heads toward traffic and then do not cross, which body heading cannot see. The LiDAR-only baseline measures how large this gap is. We include LiDAR heading as the prior in the fusion rather than throwing it away.

Why not share intermediate feature maps, as CoBEVT or V2VNet do? Feature sharing might learn the right weighting implicitly. But it costs orders of magnitude more bandwidth: DAIR-V2X reports about 0.3 KB per frame for late fusion against about 1.3 MB for raw point clouds. It also needs head-orientation labels across many paired views, which do not exist at scale in real data. Our three floats per pedestrian fit inside late-fusion budgets. If the simulated experiments show that the cone representation loses accuracy relative to an oracle trained on feature maps, that would argue for intermediate fusion. We do not run that oracle in the first phase.

Why not fill in the occluded head with a generative model, like ODM? Imputing from history cannot detect a new head turn that happens while the face is hidden. The ODM authors report invented paths under sudden behavioral changes. In our setting, imputation is the right tool only when no sensor sees the head. The fused cone's low total $\kappa$ tells the system exactly when it is in that situation.

Proposed Experiments

The first experiment is in simulation with ground truth. We extend the simulated multi-agent setup underlying OPV2V and V2XSet with scripted pedestrians whose head orientation is logged at every frame. Occluders such as buses and parked vans are placed between the pedestrian and the ego vehicle, and night lighting is varied. Infrastructure cameras and vehicle LiDAR follow the heterogeneous sensor configurations of OPV2V-H. This gives exact head-direction labels for every viewpoint, enabling measurement of angular error, cone coverage (the fraction of true directions inside the nominal 90% cone), and the area under the curve for "looked at ego." Results are broken down by face visibility, light level, and message delay.

The second experiment is a real-world check on DAIR-V2X-C. We hand-annotate head yaw for a few hundred pedestrians that appear in both the vehicle and the infrastructure images. The annotation is coarse, in eight direction bins, to keep it cheap. On this subset we test whether simulated gains in calibration and anchoring survive real extrinsic calibration error (compare Batch Differentiable Pose Refinement) and real time delays (Latency-Aware Collaborative Perception).

The third experiment is downstream intention on PIE and JAAD. These datasets are monocular, so they test only the uncertainty mechanism and not the cooperation. We swap FHPE-Net's head stream for our calibrated single-view cone and evaluate on clean frames, on synthetically darkened frames, and on the head-masked portion of the partial-occlusion segments.

Three results decide the outcome. If learned confidence does not beat fixed confidence on cone coverage, the core claim fails. If fused cones do not beat best-view selection on the face-occluded simulated subset, the cooperative claim fails and we report calibration as the whole contribution. If downstream intention F1 under darkness does not improve over retrained FHPE-Net, the signal is not worth transmitting.