When a parked truck blocks a vehicle's sensors, no amount of clever processing will reveal whether a pedestrian stands behind it. Collaborative perception gets around this by letting cars and roadside units exchange what they see over a wireless link. That only works while the link is up and fast. A natural fix is privileged distillation: train a single-vehicle "student" network to imitate a "teacher" network that saw the fused, occlusion-free view from every agent. Here privileged means the information exists at training time but not at deployment. Done naively, this asks the student to reproduce teacher features for objects it physically cannot observe. The likely result is confident hallucinations: false detections in empty shadows, and missed detections where the guess goes the other way.
We propose a more careful division of labor. The vehicle's own LiDAR, by tracing beams to see which cells they reached, splits the bird's-eye-view grid into two kinds of cells. Inside the vehicle's line of sight, the student imitates the multi-agent teacher's features. Those cells are seen, just sparsely, and the teacher's denser view gives a better target. Inside the vehicle's shadows, the student does not imitate the teacher at all. Instead a separate head is trained with a proper scoring rule to output a calibrated probability that each shadowed cell is occupied. The evidence for that probability usually sits far from the shadow: crosswalk geometry, an intersection layout, a bus stop forty meters away. So the belief head needs long-range spatial routing, not local convolutions. The claim is that this split beats both naive full-view distillation and plain label supervision on the metrics that matter for safety. It should also give a lone vehicle a usable sense of risk when collaboration fails.
Core Claim
A single-vehicle camera–LiDAR model trained against a multi-agent privileged teacher should treat visible and shadowed space differently. In visible space it should distill features. In shadowed space it should learn calibrated occupancy beliefs, conditioned on global scene context, from outcomes the other agents reveal.
We make three falsifiable predictions on OPV2V and V2X-Sim:
- Naive feature distillation over the whole grid raises the false-positive rate in shadowed free space relative to the visibility-split model, without improving recall of occluded objects.
- A belief head built from local convolutions collapses toward the dataset's average occupancy rate, while a globally routed head is better calibrated, with lower negative log-likelihood and expected calibration error on shadowed cells.
- Visible-region distillation improves detection of sparsely observed objects beyond what training on the same labels achieves.
If prediction 3 fails, the teacher contributes nothing beyond its labels, and the proposal reduces to occlusion-aware label training. That is still useful, but it is a smaller claim.
Positioning Against Prior Work
The file shows privileged distillation thriving inside a single vehicle. BEV-LGKD uses training-time LiDAR to supply foreground masks and depth targets to a camera-only student. BEVSimDet has a camera student hallucinate LiDAR-like features from a fusion teacher, correcting projection errors with local deformable attention. In both, the privileged signal comes from the same vehicle's viewpoint. They narrow a gap between sensor types, not a gap in line of sight. We supply a privilege neither has: other agents' views of the ego vehicle's shadows.
DiscoNet is the closest method and supplies our feature-imitation loss. There, a teacher fed all agents' raw point clouds trains a student that still exchanges compressed features at runtime. We remove the runtime exchange entirely. We also argue its whole-grid feature loss is the wrong objective once the student cannot see a region.
Collaborative perception motivates the problem by showing both the value of other views and their fragility. OPV2V shows fusion raising detection well above isolated driving. Where2comm uses spatial confidence maps whose low-confidence regions are precisely the shadows, and requests features only there. BEVSync reports that every method degrades once delays pass 700 ms. Our student is meant as the fallback for that regime.
M-BEV, which reconstructs features of dropped cameras from masked training, constrains the design. Its local reconstruction beat global reconstruction, because global attention pulled in noise from irrelevant views. It also struggled at the centers of missing views. We read this as a warning that global context must be routed in a structured, sparse way, not through dense attention. For that routing we reuse recurrent criss-cross attention from CCNet, which gives every cell a two-step path to every other cell at far lower memory than dense non-local blocks. We compare it against the fused axial attention of CoBEVT.
Our separate treatment of visible regions echoes Attention-Privileged Reinforcement Learning, which used simulator masks to supervise where a student should attend. We supervise which loss applies where.
Research Program
Teacher. The teacher is an intermediate-fusion camera–LiDAR detector and occupancy segmenter. It is trained on all connected agents' sensors in OPV2V and V2X-Sim, and on vehicle plus infrastructure sensors in DAIR-V2X-C.
Visibility map. For each ego frame we compute a per-cell label by casting rays from the ego LiDAR. Each bird's-eye-view cell is marked visible, partially visible, or shadowed.
Student and losses. The student sees only ego camera and ego LiDAR. It has a shared encoder and two heads. The detection head is trained with the usual detection loss plus a feature-imitation loss against the teacher, applied only on visible and partially visible cells. The belief head outputs, for each shadowed cell $c$, a probability $p_c$ that the cell contains an agent of each class. It is trained on the teacher-revealed outcome $y_c$, combined with the visible-region terms as follows:
$$ \mathcal{L} = \mathcal{L}_{\text{det}} + \lambda \sum_{c \in V} \lVert f^{S}_c - f^{T}_c \rVert^2 + \mu \sum_{c \in O} \mathrm{BCE}(p_c, y_c) $$
Here $V$ and $O$ are the visible and shadowed cell sets, $f^{S}_c$ and $f^{T}_c$ are student and teacher features at cell $c$, and $\lambda, \mu$ are loss weights. Binary cross-entropy is a proper scoring rule, meaning it is minimized by honest probabilities. In expectation it therefore drives $p_c$ toward the true chance of occupancy given what the ego vehicle can see, which is exactly the quantity a cautious planner needs.
Context routing ablation. The belief head's routing is the main architectural variable. We compare stacked local convolutions, deformable sampling as in BEVSimDet, recurrent criss-cross attention, and fused axial attention. We measure calibration as a function of how far the relevant evidence lies from the shadow.
Modality dropout. Visibility comes from LiDAR, so we also test LiDAR dropout. In that case the student must infer its own visibility map from camera depth. This shows whether calibrated shadow beliefs survive the loss of the ranging sensor.
Baselines
The naive reference point is the isolated single-agent detector with no collaboration and no distillation. On OPV2V, the OPV2V paper reports AP@IoU=0.7 of 0.406 for its no-fusion PIXOR baseline on default CARLA towns and 0.290 on Culver City. We retrain our own camera–LiDAR no-fusion model with the same backbone as the student, so that the comparison isolates training signal rather than architecture.
The upper reference is the collaborative teacher. On OPV2V, attentive intermediate fusion is reported at 0.734 [email protected] on default towns. On V2X-Sim, DiscoNet is reported at 60.3 [email protected]. We do not expect to match these; they define the gap a lone vehicle could close.
The strongest competitor for the question we actually ask is a DiscoNet-style whole-grid feature distillation into a non-communicating student. This is the obvious Gap-1 idea from the file, and we found no reported result for it on Lacuna. It uses the same teacher, data, backbone, and training budget as our method, with feature imitation applied to all cells and no belief head. We beat it if we match or exceed its occluded-object recall while showing a lower false-positive rate in shadowed free space and better calibration.
The second strong competitor is occlusion-aware label supervision. Here the single-agent model is trained directly on the fused ground-truth boxes, including objects only other agents saw, with no teacher features. We found no reported result for it on Lacuna. It shares our belief-head loss but drops visible-region distillation, so the comparison isolates the teacher's contribution. Beating it requires a gain in AP on visible but sparsely observed objects, for example those with few LiDAR returns.
The ablations are the routing variants above, matched for parameter count. They isolate whether global context is the ingredient that makes shadow beliefs informative. All tuning budgets are equal and fixed in advance, using the same number of hyperparameter trials per method on the validation split.
Why not just…?
Why not just keep collaborative perception running and accept its failures? Collaboration is strictly better when it works, and we do not propose replacing it. The issue is the fallback. BEVSync reports all methods degrading past 700 ms of delay, and Where2comm's request maps depend on a live link. We will evaluate our student as the fallback that activates when the link drops, and compare it to an isolated detector under identical dropout schedules.
Why not just train on the occluded labels and skip the teacher? This is the real threat. In shadowed cells our belief head is label training, because the teacher's output there is nearly a deterministic copy of the outcome. The proposal therefore stakes its distillation claim only on visible regions, where the teacher's denser view gives targets richer than boxes. The label-only baseline above settles this directly. If it ties us on visible-region AP, we will report that the teacher adds nothing and that the visibility-split belief head is the contribution.
Why not just treat every shadow as occupied? A worst-case occupancy assumption is safe but makes the vehicle crawl past every parked car. We test this in a closed-loop check: a fixed rule-based speed controller consumes either the all-occupied assumption or our calibrated beliefs. We compare collision rate and travel time, where the conservative rule should be safe but slow.
Why not just use dense self-attention for the belief head? CCNet reports criss-cross attention cutting FLOPs by 85% and memory 11× relative to non-local blocks. CoBEVT reports 35 FPS with sparse axial attention. M-BEV further found that unstructured global reconstruction injects noise. We include dense attention only on reduced-resolution grids as a reference.
Proposed Experiments
Main experiment. We train on OPV2V and V2X-Sim, where multiple agents reveal each other's shadows, and transfer to DAIR-V2X-C, where infrastructure provides the privileged view on real data.
Metrics. Detection AP is reported separately for visible, partially visible, and fully shadowed objects. For shadowed cells we report negative log-likelihood, expected calibration error, and Brier score. False-positive rate is measured in shadowed cells that the teacher reveals to be empty.
Decisive check 1. Prediction 1 holds if whole-grid distillation shows a higher shadowed false-positive rate than our split model at equal occluded recall.
Decisive check 2. We take the routing ablation and mask out context beyond a radius $r$ around each shadow, then plot calibration against $r$. A local model should be flat. A globally routed model should improve as $r$ grows, which would show that far context carries the signal.
Decisive check 3. This is the label-only comparison on visible, sparsely observed objects described in Baselines.
Secondary closed-loop check. We replay occluded-crosswalk and blocked-intersection scenarios in the CARLA environment underlying OPV2V. A fixed speed rule reads the belief map, and we report collisions and travel time against the all-occupied and isolated-detector controllers. This is not reinforcement learning. A fixed rule keeps the comparison about the belief, not the planner. If the beliefs prove useful, distilling them into a learned policy is the next step.
References
- BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection
- BEVSimDet: Simulated Multi-modal Distillation in Bird's-Eye View for Multi-view 3D Object Detection
- Learning Distilled Collaboration Graph for Multi-Agent Perception
- OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication
- Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps
- BEVSync: Asynchronous Data Alignment for Camera-based Vehicle-Infrastructure Cooperative Perception Under Uncertain Delays
- M-BEV: Masked BEV Perception for Robust Autonomous Driving
- CCNet: Criss-Cross Attention for Semantic Segmentation
- CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers
- Attention-Privileged Reinforcement Learning