Learning to See Into Your Own Blind Spots from Other Cars' Hindsight

A car's sensors cannot see behind a truck. Cooperative perception fixes this by having nearby vehicles and roadside units share what they see over a wireless link. That fix fails exactly when it is needed most: when packets drop, when the link lags by 100 ms or more, or when no partner is nearby. This proposal starts from a simple observation about cooperative driving logs. For almost every region hidden from one car at time $t$, the log also contains what was actually there, either because a partner vehicle saw it at time $t$ or because the car itself drove past and saw it a few seconds later. Those observations are free labels. We propose to use them to train a single car's perception model, which works from its own recent history, to predict what occupies its current blind spots. No human annotation is needed.

This is the "time-shifted" self-supervision used in robotics, where a sensor that acts later or from closer by labels a sensor that acts earlier or from farther away. Here it is moved from a single robot to a fleet. The supervising "modality" is another agent's viewpoint, or the ego car's own future viewpoint. The two sources of hindsight label different things. A partner's view at the same instant shows where moving objects are now. The ego car's future view shows only static layout, because moving objects will have moved by then. That split motivates a two-speed temporal model: a slow pathway for static structure and a fast pathway for moving objects. The result is a car that degrades gracefully when communication fails, plus a pretraining recipe that uses unlabeled cooperative logs.

Core Claim

A car's own LiDAR history contains substantial, learnable evidence about what sits in its current occlusion shadows. There are objects that were visible and then became hidden, and there is road and structure layout implied by what was seen before. Supervision drawn from partner views and from the car's own future views, restricted to cells the car could not see, extracts this evidence better than masked-reconstruction pretraining does. The reason is that the prediction target contains information that is absent from the input, whereas masking only hides information the input already has. We expect this to show up in three places: higher detection accuracy on objects in the ego car's shadow when partner features are withheld at test time, smaller accuracy loss under packet drop and latency when the model is used inside a cooperative pipeline, and better label efficiency when used as pretraining.

Relation to Prior Work

Three robotics papers supply the core principle. Demo: Learning to Perceive Long-Range Obstacles Using Self-Supervision from Short-Range Sensors labels current camera frames with future proximity-sensor readings and triples the robot's obstacle-detection range. Learning Visual Locomotion with Cross-Modal Supervision labels camera views with the proprioception recorded when the robot later steps on that terrain. A Self-Supervised Terrain Roughness Estimator for Off-Road Autonomous Driving labels earlier laser scans with later IMU shocks. It also discounts label pairs separated by large time gaps, because pose drift corrupts the spatial match between scan and shock. We reuse that time-gap discounting directly. All three papers are single-agent and label regions the robot could already see but not yet interpret. We instead label regions the car could not see at all, and we use other agents as one of the label sources.

Seeing Through the Grass is the closest single-agent precedent for occlusion. It uses foothold contacts, interpolated with Gaussian processes, to supervise a prediction of the ground hidden under vegetation. That paper also shows the limits of its signal: it needed 300 hand-labeled images when conditions shifted. Its occlusion is penetrable cover. Ours is rigid line-of-sight blocking, which no single-agent contact signal can reach.

The strongest nearby work is CooPre: Cooperative Pretraining for V2X Cooperative Perception. CooPre merges raw point clouds from all agents into one frame, randomly masks non-empty bird's-eye-view cells, and reconstructs the missing points with a Chamfer distance loss. It reports a 4% mAP gain on V2X-Real with the AttFuse fusion model. CooPre differs from our proposal on three counts. It uses single frames. It masks cells at random rather than targeting the ego car's true occlusion shadows. Its fused input already contains the partner's view, so it teaches the model to complete a scene, not to infer hidden content from the ego car's own past.

Supervised temporal cooperative models constrain the design. Latency-Aware Collaborative Perception (SyncNet) forecasts a partner's current features from a buffer of stale ones using a pyramid of recurrent layers. Leveraging Temporal Contexts to Enhance Vehicle-Infrastructure Cooperative Perception (CTCE) reconstructs dropped roadside queries with a constant-velocity Kalman filter. Both are trained on human-labeled boxes, and both still depend on the partner's past transmissions, so they cannot help when a partner was never connected. Our model uses only ego sensing at inference.

SlowFast Networks for Video Recognition supplies the two-rate architecture: a sparse, high-capacity pathway for appearance and a dense, thin pathway for motion. We use it because our two label sources separate cleanly into static and dynamic targets, not because a two-stream design is valuable in itself. An ablation tests whether it matters.

Research Program

The first step is building targets. For each logged frame of a cooperative sequence, we ray-cast the ego LiDAR sweep into a bird's-eye-view grid, a top-down metric map of cells around the car. Each cell is labeled observed-free, observed-occupied, or occluded. For occluded cells there are two target sources. The first is the partner's same-time sweep, transformed with the logged relative pose, which gives current occupancy, including moving objects. The second is the ego car's own sweeps from later in the sequence, transformed back to time $t$. These give static-layout targets only: ground, curbs, parked vehicles, and buildings. We keep only cells that stay consistently occupied or free across the future window, which filters out moving objects. Each target cell gets a weight that decays with pose uncertainty and with the time gap, following Stavens and Thrun's discounting, and cells no source observed are left out of the loss. The training objective is a weighted binary cross-entropy restricted to shadow cells:

$$ \mathcal{L} = \sum_{c \in S_t} w_c \,\big[\mathrm{BCE}(\hat{o}^{\,dyn}_c, y^{\,p}_c) + \mathrm{BCE}(\hat{o}^{\,sta}_c, y^{\,f}_c)\big]. $$

Here $S_t$ is the set of ego-occluded cells at time $t$. $\hat{o}^{\,dyn}_c$ and $\hat{o}^{\,sta}_c$ are the model's dynamic and static occupancy predictions for cell $c$. $y^{\,p}_c$ is the partner-derived label, $y^{\,f}_c$ is the future-ego label, and $w_c$ is the confidence weight. Each term is dropped when its label is unavailable.

The second step is the model. A standard pillar-based LiDAR encoder produces bird's-eye-view features for each ego frame. A slow pathway takes a few frames spread over roughly two seconds at full channel width and predicts static occupancy. A fast pathway takes every frame over the same window at reduced width and bird's-eye-view resolution and predicts dynamic occupancy. Lateral connections link the two, as in SlowFast. All frames are warped into the current ego pose before fusion. The predicted shadow features are then added to the ego feature map that goes into any downstream detector.

The third step is deployment. We test two modes. In ego-only mode, the car has no partner, and the shadow predictor's features feed a single-car detector. In cooperative mode, the predictor sits inside an existing fusion model such as AttFuse or CoBEVT, run through the taco-group/STAMP codebase. When partner features are dropped or stale, the fusion model falls back on the ego car's own shadow prediction.

As a secondary experiment, we test a camera-only ego car supervised by partner and infrastructure LiDAR on DAIR-V2X. This checks whether the hindsight signal also carries across sensor types, as in the robotics precedents.

Baselines

The naive reference is a single-car temporal LiDAR detector trained from scratch with the same encoder and the same frame history, but no shadow supervision and no pretraining. It shows how much temporal context alone already recovers. DAIR-V2X gives the scale of the gap we are trying to close: vehicle-only LiDAR detection reaches 31.33 AP, against 41.90 AP for late fusion and 50.03 AP for early fusion on VIC-Sync (DAIR-V2X).

The strongest existing pretraining competitor is CooPre, run with its public protocol on the same unlabeled sequences, the same encoder, the same pretraining compute measured in GPU-hours, and the same fine-tuning budget. It reports a 4% mAP improvement with AttFuse on V2X-Real. We would count it as beaten if our method matches or exceeds it on overall cooperative mAP and clearly exceeds it on shadow-region AP and under partner dropout. A stricter version of this baseline, multi-frame CooPre, masks random cells in stacked multi-frame fused clouds. It shows whether any gain comes from temporal context rather than from the shadow-targeted objective.

The strongest cooperative detector in real data is CoBEVT, which reaches 66.5% [email protected] on V2V4Real (V2V4Real). We run it with and without our shadow module under matched training. For robustness to lost communication, the reference points are CTCE, which reports 50.4% mAP on V2X-Seq with only a 3.4% drop at 50% packet loss (CTCE), and SyncNet, which reports +15.6% AP over DiscoNet under latency on V2X-Sim. Beating these means a smaller relative accuracy loss than they report at matched drop rates and latencies.

For detection restricted to objects in the ego car's occlusion shadow on real data, no reported result was found on Lacuna. We will compute this metric for every baseline ourselves.

Two ablations isolate the key ingredient. The first keeps the architecture and data fixed but moves the loss onto randomly masked visible cells instead of true shadow cells. This separates the benefit of the hindsight target from that of simply reconstructing masked input. The second replaces the two-rate network with a single-rate network of matched compute, which tests whether the static/dynamic split earns its place.

Why not just…?

Why not just train a temporal ego detector on the cooperative labels, which already include occluded objects? This is the most serious threat. V2V4Real and DAIR-V2X annotations cover objects the ego car cannot see, so with full labels, supervised training already pushes the model to infer hidden objects. We do not expect to beat it clearly at 100% labels on box AP. Our claims are that hindsight pretraining matches it with a fraction of the labels, in the way CooPre matched full-data training with 50% of the labels, and that dense occupancy targets also cover static layout, which box labels do not. The label-fraction sweep will settle this. If supervised training on 10% of the labels matches our pretraining plus 10% of the labels, the pretraining claim fails.

Why not just carry tracks through occlusion with a Kalman filter? Track persistence handles objects that were seen and then became hidden, provided they move predictably. It cannot place objects the car never saw. CTCE's authors rely on exactly this constant-velocity recovery, and it fails during non-linear maneuvers. We split shadow-region evaluation into objects previously visible and objects never visible, and we run a tracker-persistence baseline on the first group. If tracking matches us there, the value of the learned model rests on the never-visible group and on static layout. We will report that honestly.

Why not just use CooPre or multi-frame masking? Random masking of fused clouds rewards completing structure that neighboring visible points already imply. Our target, by construction, lies outside what the ego car sensed. The shadow-versus-random-mask ablation, run with the same network and data, answers this directly.

Proposed Experiments and Decisive Checks

We pretrain on unlabeled sequences from V2V4Real, OPV2V, and DAIR-V2X-C, then fine-tune detection at 10%, 25%, 50%, and 100% of the labels. Evaluation uses V2V4Real, DAIR-V2X VIC-Sync, and V2X-Seq, with cross-dataset transfer from V2V4Real to V2X-Real.

The metrics are [email protected] and [email protected] overall. We add AP restricted to boxes whose centers fall in ego-occluded cells, split into previously-visible and never-visible objects. We also report occupancy intersection-over-union on shadow cells, broken down by static and dynamic content. Robustness curves come from dropping 0–80% of partner packets and injecting 0–300 ms of latency, applied identically to all methods. As a sanity check on the targets, we measure how much target cells disagree with human box labels, and plot it against pose error and time gap. The decay weights should visibly reduce this disagreement.

Three results would decide the proposal. First, with partner features withheld at test time, the shadow-supervised ego model should improve AP on the previously-visible occluded subset by a clear margin over the naive temporal detector. If it does not, the ego car's history carries too little signal and the premise is wrong. Second, the shadow-target objective should beat the random-mask ablation at equal compute. If it does not, the contribution reduces to multi-frame CooPre. Third, inside CoBEVT under 50% packet loss, adding the module should shrink the accuracy drop relative to CoBEVT alone. A negative result on the two-rate ablation would be informative rather than fatal, and we would then drop the slow/fast design in favor of the simpler network.