Many robots and cars carry a spinning LiDAR that measures depth all the way around them, paired with a camera that sees only a forward cone. Models that predict 3D occupancy, meaning which cells of a 3D grid around the vehicle are free, occupied, and by what kind of object, therefore get rich color and texture evidence ahead and only sparse geometry to the sides and rear. Current methods either invent camera features for those regions or label them "unobserved." This proposal starts from a simple observation. A vehicle driving forward has usually already seen, with its front camera, most of what is now beside and behind it. During training, where past and future frames and accurate poses are available offline, we can carry the camera's evidence forward and backward in time. We anchor it to LiDAR geometry, place it in the peripheral voxels of the current frame, and distill it into a student network that at test time sees only the current frame.
The claim is that this training-only "hindsight teacher" moves camera-quality semantics into the LiDAR-only part of the volume at zero inference cost. It should give its largest gains in the label-scarce regime and in a band around the camera's frustum edge, which is where asymmetric fusion is known to produce artifacts. The work also contributes a coverage-stratified evaluation that splits occupancy accuracy into three zones (camera and LiDAR, LiDAR only, neither) so that peripheral failures stop being averaged away.
Where the gap is
Work on asymmetric coverage has handled the side and rear of the vehicle in three ways, and none of them gives peripheral voxels a grounded camera-derived training signal. In the first approach, cross-modal distillation such as xMUDA transfers knowledge between 2D and 3D branches only where both see the same point. Rendering-based self-supervision such as Multi-modal NeRF Self-Supervision casts camera rays through a LiDAR voxel grid and reports +5.0% mIoU on nuScenes with 1% labels, but its rays cannot leave the camera frustum, so voxels outside it get no rendering gradient. UGOCC checks pseudo-labels by rendering occupancy into camera planes and has the same restriction. These papers motivate our work, because they show that camera-derived supervision helps 3D learning wherever it can reach.
In the second approach, MSeg3D adds a module that predicts "pseudo-camera" features from LiDAR outside the camera cone. That module learns only from in-frustum pairs, so out-of-frustum predictions are extrapolation with no target. In the third approach, MR-Occ and Occ3D model occlusion or visibility explicitly as a voxel state. That choice is honest, but it gives up on recovering semantics there. Our proposal resolves the tension between completion and occlusion modeling in a specific way. Peripheral voxels get camera-derived targets only where the camera actually observed them at some other time, and they stay "unobserved" otherwise.
The closest related work is C3DFusion. It warps past camera-derived 3D points into the current frame at inference time to fill out-of-view regions and reaches 44.37 out-of-view IoU on SemanticKITTI against roughly 30–33 for earlier temporal models. However, its performance peaks at 4 frames because monocular depth errors and pose drift accumulate. Our proposal differs on three axes. It runs only at training time, so it can use future frames and offline-optimized poses. It places camera evidence using LiDAR depth instead of estimated depth. And it is a camera+LiDAR method rather than camera-only. These differences are exactly why we expect the hindsight teacher to keep improving past C3DFusion's 4-frame ceiling. That expectation is tested directly.
The volumetric-context literature constrains the student design. 3D U-Net showed that 3D convolutions trained on sparsely annotated slices produce dense volumetric predictions: 0.863 IoU against 0.796 for a 2D network on its microscopy data. This is the precedent for expecting supervision on a partial, scattered subset of voxels to spread through native 3D context. Hindsight targets will cover only some peripheral voxels, so the student must be a native 3D occupancy network, not a bird's-eye-view or slice-based one.
Core mechanism
For a training frame at time $t$, we gather frames in a window $[t-T, t+T]$. For each of them we run a camera-informed teacher inside its own camera frustum. The teacher is the same occupancy network, or an EMA copy of it, applied with its in-frustum camera features. We lift its voxel features and class probabilities into the frame-$t$ grid using LiDAR-measured geometry and offline poses. We keep only static voxels, identified by LiDAR scene flow or multi-sweep consistency. Each target voxel is then aggregated across contributing frames with a confidence weight that decays with range from the camera and with residual LiDAR–camera disagreement.
The student sees only frame $t$, with the front camera and the full LiDAR sweep. It is trained with its usual occupancy loss plus a distillation term. That term applies only to voxels outside the current camera frustum that received hindsight evidence:
$$ \mathcal{L}_{\text{hind}} = \sum_{v \in \mathcal{P}_t} w_v \Big[ \mathrm{KL}\big(q_v \,\|\, p_v\big) + \lambda \,\lVert f_v - g_v \rVert_2^2 \Big] $$
In this loss, $\mathcal{P}_t$ is the set of peripheral static voxels with hindsight coverage at time $t$. The weight $w_v$ is that voxel's aggregated confidence. $q_v$ and $g_v$ are the teacher's class distribution and feature for voxel $v$, and $p_v$ and $f_v$ are the student's. $\lambda$ balances logit and feature matching. Voxels with no hindsight coverage get no distillation term and keep their explicit "unobserved" supervision, so the model is never asked to invent semantics that no camera ever measured.
Research Program
The first step is a feasibility measurement that needs no training. On nuScenes and SemanticKITTI we compute the hindsight coverage fraction: the share of occupied peripheral ground-truth voxels that fall inside the front camera frustum at some frame within $\pm T$, as a function of $T$ and of driving maneuver (straight, turning, stopped). If this fraction is low except on straight driving, the method's ceiling is low, and we report that as the main finding.
The second step builds a controlled asymmetric rig. Occ3D-nuScenes and nuScenes-Occupancy have six surround cameras and dense labels everywhere. We restrict the student to the front camera only, while the full 32-beam LiDAR and the 360° labels stay available. This gives exact ground truth in the zones the student cannot see with a camera. It also gives a privileged upper bound: a teacher that uses the five withheld cameras at the same timestamp. SemanticKITTI then serves as a naturally asymmetric rig, with a 90° front camera and a 360° LiDAR, where no such privileged cameras exist. That checks the method without the synthetic restriction.
The third step trains one fixed student architecture, an MR-Occ-style camera-LiDAR occupancy network, under every training recipe compared below. Label budgets are 1%, 10%, and 100%, with identical schedules. Differences can then be attributed to the training signal rather than the architecture.
The fourth step is analysis. We report accuracy in each coverage zone, in a boundary band within a few degrees of the frustum edge, and by distance. We also report calibration (expected calibration error) of peripheral class probabilities, since the MSeg3D-style hallucination drift that motivates this work should appear as overconfidence there.
Baselines
The naive reference point is the student trained with its standard occupancy loss under the front-camera rig. Missing camera features outside the frustum are zero-filled and a binary coverage channel is appended. This is the decoration scheme critiqued in PointAugmenting. We train it with random camera-dropout augmentation, which makes it a fairly strong cheap baseline rather than a straw man. A LiDAR-only version of the same network is reported alongside it as the floor for the LiDAR-only zone.
The strongest existing occupancy method is MR-Occ. On nuScenes-Occupancy with all six cameras it reports 35.5% IoU and 27.3% mIoU (MR-Occ). No reported result was found on Lacuna for MR-Occ, or any occupancy model, under a front-camera-only nuScenes rig, so we retrain it under that rig with its published recipe and the same label budget and epochs as our student. Because our student shares its architecture, this run is also the "no hindsight" ablation.
The strongest competitor for the specific problem of filling out-of-frustum voxels is learned feature completion. We transplant MSeg3D's cross-modal feature completion module into the same student, with the same data and tuning budget. If completion matches hindsight distillation in the LiDAR-only zone, our extra supervision is unnecessary.
For inference-time temporal context we add a C3DFusion-style memory to the student, warping the past four frames of camera-derived features with LiDAR depth. On SemanticKITTI we report the published camera-only reference of 44.37 out-of-view IoU (C3DFusion) for context, noting that it is not a like-for-like comparison with a LiDAR-equipped model.
Ablations isolate the key ingredient. We compare a past-only hindsight window, which uses only information an online memory could in principle have, against the full past-and-future window. We compare LiDAR-anchored lifting against lifting with estimated monocular depth. We compare static-only distillation against distillation that includes dynamic objects warped by ground-truth box tracks (an oracle). Finally, we compare against the privileged five-camera teacher (the upper bound).
Beating the baselines means a statistically reliable gain in LiDAR-only-zone mIoU over both the retrained MR-Occ and the MSeg3D-completion variant at the 1% and 10% label budgets, measured across three seeds. It also requires no loss of in-frustum mIoU and a lower peripheral calibration error.
Why not just…?
Why not just train on the dense occupancy labels, which already cover the periphery? On fully labeled nuScenes this is a real threat. Ground-truth labels in the side and rear zones already supervise the student, and hindsight distillation may add little at 100% labels. Our claim is aimed at the label-scarce regime, where Multi-modal NeRF Self-Supervision showed camera-derived self-supervision is worth several mIoU points inside the frustum. The label-budget sweep settles this directly. We predict the gain shrinks as labels grow, and if it is near zero even at 1% the idea fails.
Why not just append a coverage mask and train with camera dropout? This tells the network where it is blind but gives it nothing to learn about what is there. SGFormer reports 2–10% robustness gains from adaptive fusion over static fusion under limited field of view and sensor dropout, which suggests masking-style tricks leave headroom. The masked, dropout-augmented student is our naive baseline. If it matches us in the LiDAR-only zone, the hindsight signal is redundant.
Why not just use temporal memory at inference, as C3DFusion does? Online memory cannot use future frames and pays inference cost. Its reported ceiling at 4 frames (C3DFusion) comes from depth and pose drift that offline LiDAR anchoring should avoid. That said, LiDAR-anchored online memory may close much of the gap. We run it under identical conditions and also combine it with hindsight distillation. If the memory alone matches the combination, we will report that hindsight's value lies only in its zero inference cost.
Why not just use MSeg3D's feature completion? Completion is trained only where both sensors overlap, then applied where they do not. That is extrapolation without a target, and the source of the hallucination drift noted for MSeg3D. The transplanted completion baseline and the calibration metric test whether real hindsight targets produce better-grounded, better-calibrated peripheral predictions.
Decisive checks
The idea survives only if four things hold. First, the hindsight coverage fraction is substantial, roughly a majority of peripheral occupied voxels for windows of a few seconds in ordinary driving; otherwise the method has little to work with. Second, at 1% and 10% labels the hindsight student beats retrained MR-Occ and the completion variant in LiDAR-only-zone mIoU and in the frustum-boundary band, and closes a meaningful fraction of the gap to the privileged five-camera teacher. Third, the horizon sweep separates the two lifting schemes: LiDAR-anchored lifting keeps improving or plateaus well past 4 frames, while monocular-depth lifting degrades near C3DFusion's reported limit. A curve like that would show that LiDAR geometry, not the extra frames alone, is what pushes past the 4-frame ceiling. Fourth, the gains carry over to SemanticKITTI, where the asymmetry is real rather than simulated.
If the past-only window recovers nearly all of the benefit, the practical conclusion shifts toward online LiDAR-anchored memory, and we will say so.
References
- xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
- Multi-modal NeRF Self-Supervision for LiDAR Semantic Segmentation
- Enhancing Generalizability via Utilization of Unlabeled Data for Occupancy Perception
- MSeg3D: Multi-Modal 3D Semantic Segmentation for Autonomous Driving
- MR-Occ: Efficient Camera-LiDAR 3D Semantic Occupancy Prediction Using Hierarchical Multi-Resolution Voxel Representation
- Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
- Towards Temporal Fusion Beyond the Field of View for Camera-based Semantic Scene Completion
- 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
- PointAugmenting: Cross-Modal Augmentation for 3D Object Detection
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation