A camera-only car can learn a 3D map of what occupies the space around it without anyone labeling that space in 3D. The method is to render its 3D guess back into 2D depth and semantic images and compare those renders against 2D targets. The strongest versions of this idea borrow rays from neighboring video frames. Those extra viewpoints see the far side of a parked car or the area behind a pole, which fills in geometry a single snapshot cannot see. The catch is that borrowing rays across time assumes the world holds still. A moving car seen half a second later appears to be somewhere else, so the network learns smeared "ghost" cars. The only method that fixes this properly, OccFlowNet, moves each car's predicted voxels to where the car will be before rendering. To know where to move them, it needs ground-truth 3D bounding-box trajectories, which are the expensive 3D labels this whole line of work was trying to avoid.
We propose to remove that dependency. The network predicts occupancy at every frame anyway. So we can estimate each moving object's motion by rigidly registering its predicted 3D blob at time $t$ to its predicted blob at the neighboring time $t'$. That motion then warps the object before the neighboring frame's rays are rendered. When registration is unreliable, a confidence gate falls back to discarding that object's borrowed rays, which reduces to today's filtering baseline. The claim is concrete. On Occ3D-nuScenes, under the same 2D-only supervision as OccFlowNet, this self-bootstrapped motion should recover most of the gap between OccFlowNet without flow (26.14 mIoU) and with box-derived flow (28.42 mIoU), and it should use no 3D boxes anywhere in training. Because correctly warped neighbor-frame rays reach the unseen sides of moving vehicles, it should also complete the hollow back halves of dynamic objects, which filtering-based methods cannot do.
Where This Sits
Three rendering-supervised occupancy methods define the current trade-off. RenderOcc introduced auxiliary rays from adjacent frames. With 2D labels alone it reaches 23.93 mIoU, but it treats moving objects as noise, handled only through a class-balancing ray-sampling heuristic. UniOcc aggregates temporal views but explicitly filters dynamic objects out of them, so it recovers static background and never completes moving agents. SelfOcc goes fully self-supervised with a signed distance field and photometric reprojection, but it assumes a static scene within its supervision window and blurs on moving agents. These three papers motivate the proposal: temporal rays are the source of completion, and dynamics are the source of corruption.
OccFlowNet supplies the method we reuse. It warps predicted dynamic occupancy to adjacent timestamps before volume rendering, which gains 10.05 points on dynamic classes over naive filtering. It also sets the bar we want to reach without its box labels. Its rigid-box motion model also limits our design, since we keep rigid per-object motion and handle deformable agents by gating rather than pretending to model them.
Two neighboring lines of work also inform the design. TraPCC completes hollow vehicle shells by accumulating tracked LiDAR points over time. That confirms temporal accumulation is what fills backsides, but it needs a tracker and works on raw points with no image supervision. Split4D separates dynamic from static Gaussians without video segmentation by giving primitives velocities. It optimizes one scene at a time and has no completion prior, whereas we want a feed-forward predictor trained across a dataset.
EnerGS contributes a framing. It splits space into occupied, free, and unknown regions from line-of-sight evidence, and its static assumption breaks precisely because free-space evidence is time-stamped. Our warping step makes that time-stamping explicit for the objects that move.
Core Claim
Box-derived flow in OccFlowNet does one job: it tells the renderer where a moving object's voxels are at time $t'$. The network's own occupancy prediction at $t'$ already contains a noisy answer to that question. We extract connected components of dynamic-class voxels from the prediction at $t$ and at $t'$, register each component with a rigid transform, and use the transform to warp density and semantics before rendering rays from $t'$. With a confidence gate, this yields most of the dynamic-class benefit of ground-truth flow.
This differs from all prior work in two ways. The motion signal comes from the model's own predictions, not from 3D annotations or 2D trackers. And the fallback is principled: a component whose registration residual exceeds a threshold has its auxiliary rays masked, so the method cannot do worse than filtering on the objects it cannot explain.
Research Program
Base model. We use OccFlowNet's architecture unchanged: a Swin Transformer backbone, BEVStereo lifting from 2D image features to 3D, and a 3D CNN decoder that predicts density and semantic logits per voxel. We also use its 2D depth and semantic targets, which are projected LiDAR and 2D segmentation. Keeping these fixed means that any difference comes from where the motion signal originates.
Training. Training runs in two phases.
- Warm-up. The model trains with current-frame rays plus filtered auxiliary rays until its predictions on dynamic classes stabilize.
- Self-bootstrapped flow. For each training sample, the network runs on frames $t$ and $t'$ with gradients stopped. We extract dynamic-class components at each time, match them across time by semantic class and proximity using ego-motion-compensated coordinates, and register matched pairs with a rigid transform. The registration minimizes a density-weighted point-to-point objective over voxel centers, initialized from the centroid offset. We then warp the time-$t$ prediction by the estimated transform and render the auxiliary rays from $t'$, exactly as OccFlowNet does with box flow.
Confidence gate. The gate uses three signals: the registration residual, the mismatch in component volume, and component size. Small, non-rigid-looking components, which are typically pedestrians, fail the gate by default.
Studying the flow directly. Because flow estimates improve as occupancy improves and vice versa, we test whether this loop converges or drifts. We log registration error against ground-truth box motion throughout training. Boxes are used only for this diagnostic and never for training.
Label-free variant. As a secondary study, we port the same mechanism into a SelfOcc-style photometric pipeline. This tests whether the idea survives when the 2D targets are not derived from LiDAR. That variant reports geometric IoU only, since semantics there would still need supervision.
Baselines
Naive reference points. The first is OccFlowNet's own base configuration: 2D-only supervision with no flow and dynamic objects filtered, which the paper reports at 26.14 mIoU on Occ3D-nuScenes. The second is a current-frame-only variant with no auxiliary rays at all. It isolates how much temporal rays help before dynamics enter the picture. We will retrain both in our codebase with the same backbone, epochs, and number of auxiliary frames, so that reproduction drift is measured rather than assumed.
Strongest existing method. This is OccFlowNet with ground-truth box flow, reported at 28.42 mIoU under 2D-only supervision OccFlowNet. It is the strongest competitor because it solves exactly our problem, but with privileged 3D labels, so we treat it as an oracle upper bound. It will be rerun with identical data, schedule, and tuning budget. "Beating it" is not the goal. Success means recovering at least two-thirds of the 26.14-to-28.42 gap overall, and a comparable share of its dynamic-class gain, with zero 3D labels. Matching it would be a strong result.
Ablations that isolate the key ingredient. - Dense flow instead of rigid registration. We remove registration and let a per-voxel flow head learn from the rendering loss. This tests whether rigid grouping of the network's own predictions is what carries the method, or whether a flexible flow field trained end-to-end is enough. - No gate. We run without the confidence gate to measure how much the fallback protects against bad registrations.
Why not just…?
Why not just run a pretrained 3D detector and tracker to produce pseudo-boxes, then train OccFlowNet on those? This is the most serious threat, and we will run it. The limitation is structural rather than empirical: such a detector was itself trained on 3D box labels, so the pipeline still consumes 3D annotation, only indirectly. We report it alongside our method. If pseudo-boxes clearly beat self-registration, our contribution narrows to the strictly label-free setting, and we will say so. If the two are close, our method removes an entire external model and its failure modes, including the tracker drift that Split4D documents, at no cost.
Why not just mask dynamic pixels with a 2D segmenter and drop their borrowed rays, as UniOcc does? Masking removes ghosts, but it also removes the only views that could complete moving objects. OccFlowNet reports that motion-aware warping beats naive filtering by 10.05 points on dynamic classes. Our gated method contains filtering as its fallback, so the comparison directly measures what successful registration adds. It is especially informative on dynamic voxels that are invisible from the current frame's cameras.
Why not just learn a per-voxel flow field end-to-end from the rendering loss? Nearby cars move meters between the sampled frames. A gradient flowing through trilinear warping only sees local displacements, so it is unlikely to find motions that large. This alternative is the dense-flow ablation above. If it matches rigid registration, the simpler design wins and we adopt it.
Why not just accumulate points over time for moving cars, as TraPCC does? TraPCC reports that accumulation hits a sparsity floor for distant or parallel-moving vehicles. It also needs tracks and gives no image-based supervision. It addresses offline shape completion, not a camera-only occupancy predictor, so we cite it as motivation rather than run it as a baseline.
Proposed Experiments
Headline comparison. The main benchmark is Occ3D-nuScenes, which covers 80m × 80m × 6.4m at 0.4m voxels. We report overall mIoU and dynamic-class mIoU under the camera visibility mask. We compare our method against both OccFlowNet configurations, the pseudo-box pipeline, and the ablations. For context, fully 3D-supervised BEVStereo reaches 24.51 mIoU as reported by RenderOcc, and the rendering-supervised UniOcc reaches 52.1 mIoU on validation in its challenge configuration UniOcc. The UniOcc figure uses a far stronger setup and is not a like-for-like target.
Decisive checks. - Hollow-shell completion. We evaluate dynamic-class IoU restricted to ground-truth-occupied voxels that are not visible from the current frame's cameras but are covered by the LiDAR mask. This is where warped neighbor-frame rays should help and filtering cannot. If our method shows no gain here over filtering, the completion claim fails. - Ghost volume. We count predicted dynamic-class voxels lying in ground-truth free space along object trajectories. - Speed stratification. We bin objects by speed. Gains should concentrate on fast movers, and pedestrians should mostly fall back to filtering. - Registration accuracy. We report translation and rotation error of registration against ground-truth box motion over the course of training. If this error does not fall during training, the bootstrap is not working, whatever mIoU says.
No reported result was found on Lacuna for any box-free occupancy-flow method on Occ3D-nuScenes, so these experiments establish the first such numbers.
References
- OccFlowNet: Occupancy Estimation via Differentiable Rendering and Occupancy Flow
- RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
- UniOcc: Unifying Vision-Centric 3D Occupancy Prediction with Geometric and Semantic Rendering
- SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction
- Self-Supervised Point Cloud Completion on Real Traffic Scenes Via Scene-Concerned Bottom-Up Mechanism
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- EnerGS: Energy-Based Gaussian Splatting with Partial Geometric Priors