Supervising Moving Objects Through the Background They Hide

Self-supervised 3D perception learns depth and occupancy from video by checking whether the predicted geometry can redraw one frame from its neighbors. Moving cars and pedestrians break that check, so the standard fix is to mask them out of the loss. The result is that the network gets no geometric supervision on exactly the objects a driving system most needs to locate. The documented failures follow from this. A car that keeps pace with the ego-vehicle gets pushed to infinite depth (Kumar et al.), and crossing traffic comes out "sunken" into the road or floating above it (Moon et al.).

This proposal starts from a simple observation. Even when we cannot say how an object moved, we can say a lot about where it is at a single instant, because it is hiding a static scene we have already reconstructed from other moments. If a pixel shows a moving car, the car must sit in front of the background that the time-aggregated static map places along that pixel's ray. It must also rest on the ground surface the map recovers beneath it, and it cannot sink below that surface. Both statements are constraints at a single timestep. They need no motion model, no optical-flow network, no segmentation model and no 3D boxes. We propose to turn them into training losses inside a height-preserving self-supervised occupancy model. We will then test whether they recover dynamic-object geometry that masking throws away.

Core Claim

A self-supervised occupancy network can learn metric geometry for moving objects without modeling their motion. It does this by supervising each moving pixel with two bounds derived from a static map of the scene. The first is an occlusion upper bound: the object lies no farther than the background it hides. The second is a ground-contact bound: the object's lowest occupied cell touches, and does not penetrate, the map's ground surface in that column.

The two bounds directly target the two documented failure modes. Infinite-depth and sunken predictions violate the occlusion bound. Floating predictions violate the contact bound. The contact bound is only expressible if the representation keeps height. In a flat top-down grid, the object's base and the road beneath it occupy the same horizontal cell, so the boundary between them collapses (Khaled Saleh et al.). Height preservation is therefore a precondition of the method, not a separate contribution.

Relation to Prior Work

Several papers supply motivation. SelfOcc learns signed-distance occupancy from surround video by rendering depth and color, but it assumes the world is static within its rendering window and blurs moving traffic. BEVScope fuses surround cameras through a flat top-down grid, which discards height. It also downweights high-motion pixels using a structural-similarity signal, so those pixels receive no supervision. Both papers show the gap: dynamic regions are either corrupted or silenced.

Three papers attack the gap by modeling motion, and each pays a price in external supervision. OccFlowNet warps dynamic voxels between frames using flow computed from ground-truth 3D boxes, and reports a 10.05% dynamic-class mIoU gain over naive filtering. That result is evidence that dynamic supervision matters, but it is not self-supervised. Let Occ Flow learns occupancy and 3D flow on three orthogonal planes, and leans on Unimatch optical flow and GroundedSAM masks. Its accuracy is bounded by those models. Split4D (Hu et al.) separates static and dynamic components during optimization. However, the file notes that motion-geometry decoupling without masks stays ill-conditioned when camera and object motion are collinear. Our constraints avoid this problem because they never estimate motion.

The closest single paper is From-Ground-To-Objects. It anchors moving objects to the road using a disparity-smoothness loss at the bottom edge of Mask2Former instance masks, and reports a 17.0% Abs Rel improvement over MonoViT on Cityscapes. We keep its physical insight but change three things. First, the contact constraint lives in 3D occupancy rather than image disparity. Second, dynamic regions are found by disagreement with the static map rather than by a 2D segmentation model. Third, we add the occlusion bound, which it lacks.

Two further papers supply method. SliceOcc supplies the height-preserving representation: horizontal slices linked by cross-attention. Its slice-count ablation (12.13% to 14.57% mIoU going from 2 to 8 slices on EmbodiedScan) is evidence that height resolution matters. It is, however, supervised and indoor-static, and we use it self-supervised and outdoors. RobustSplat supplies the logic for building a clean static map. Transient content produces residuals that are inconsistent over time and can be kept out of the persistent model. RobustSplat discards those transients. We use them as the supervisory signal.

Research Program

The model is a SelfOcc-style signed-distance occupancy network. Its top-down or three-plane lifting is replaced by vertical slices, and it is trained with the usual rendered photometric loss on pixels judged static.

During training, the model maintains a static map per sequence. To build it, we aggregate the network's own predicted occupancy over the sequence in the world frame using the estimated ego-poses. We keep only cells that remain occupied or free consistently across time, so transient objects fall out. The map is refreshed periodically, the way pseudo-labels are refreshed in self-training.

At each frame, we render depth from the static map for every pixel. A pixel is treated as foreground at time $t$ when the observed image disagrees with the static rendering beyond a per-image statistical threshold, in the spirit of Jiang et al.. We call the resulting soft weight $w(p)$.

For foreground pixels, the ordinary photometric loss stays masked, as in prior work. Two new terms are added instead. The occlusion term is

$$\mathcal{L}_{\text{occ}} = \sum_p w(p)\,\max\big(0,\; \hat d(p) - D_s(p) + \delta\big),$$

where $\hat d(p)$ is depth rendered from the current prediction, $D_s(p)$ is depth rendered from the static map, and $\delta$ is a small margin.

The contact term is defined on predicted occupied cells that are absent from the static map. These are grouped into connected components. For each component, the term penalizes the vertical gap between its lowest cell and the map's ground height in that column, in both directions. This rules out both floating and sinking. It is soft, and it is gated off above a clearance threshold so that genuinely overhanging structures are tolerated.

There is a useful special case. For a vehicle that keeps pace with the ego-vehicle, the rays through its bottom rows hit road that the static map recovers from later frames, after the car has moved on. The occlusion bound alone therefore already pins the car's depth to near the correct value. This is the case where photometric self-supervision currently fails.

The analysis has three parts. First, we measure how tight the bounds are in practice, by comparing $D_s$ against LiDAR on foreground pixels. Second, we measure how often the self-built map absorbs objects that stop temporarily. If a stopped car enters the map, it is supervised as static, which is correct for that interval, but the map must release it when the car leaves. Third, we test whether errors in the self-built map feed back into the training loop. To bound that risk, we include an oracle variant that builds the map from LiDAR.

Baselines

The naive reference point is the same network trained with no dynamic handling. Every pixel receives the photometric loss under a static-world assumption, as in SelfOcc. Its reported numbers are 45.01 IoU on nuScenes surround occupancy and 36.27 RayIoU on SemanticKITTI (the latter as reported by Let Occ Flow). We retrain it on our backbone and schedule so that only the loss differs.

The masking baseline zeroes the loss on foreground pixels using the same $w(p)$ and adds nothing else. This is the BEVScope-style choice. It isolates whether the bounds add supervision beyond simply protecting the background. BEVScope reports 0.232 Abs Rel on nuScenes surround depth, against 0.240 for SurroundDepth and 0.237 for MCDP. We report against these figures, but we compare primarily against our own retrained masking variant, because the architectures differ.

The strongest self-supervised competitor is Let Occ Flow, which reports 47.06 RayIoU on SemanticKITTI and 1.45 mAVE on nuScenes. It is the strongest because it does supervise dynamic regions, but it uses Unimatch flow and GroundedSAM. We run it with its released recipe on the same splits and training budget. We also run a variant of our model that adds its external-model flow loss, so we can see whether the two sources of supervision are complementary.

The ported prior baseline is a 3D version of From-Ground-To-Objects: its ground-contact loss driven by Mask2Former masks inside our network, with no occlusion term.

As an upper reference, not a competitor, we include OccFlowNet, which uses ground-truth boxes and LiDAR. It reports 28.42% mIoU on Occ3D-nuScenes, versus 26.14% for its 2D-only base and 23.93% for RenderOcc.

A further baseline distills pseudo-depth on foreground pixels from a large pretrained monocular depth model of the learned-prior kind (Liu et al.). Each frame's pseudo-depth is scale-aligned to the static map.

The ablations isolate the key ingredients: occlusion only, contact only, both together, and representation swapped among flat top-down, three-plane and slices. The representation swap tests whether height preservation is really needed for the contact term.

All methods share the backbone, image resolution, number of epochs and a fixed hyperparameter search budget. We have no reported result on dynamic-pixel-only depth error for any of these methods; no reported result was found on Lacuna for dynamic-region Abs Rel on nuScenes, so we will compute it for every baseline ourselves. Beating the field means two things. On dynamic pixels, we need lower Abs Rel than the masking variant and the 3D ground-contact port, with confidence intervals across three seeds. On dynamic classes, we need occupancy mIoU on Occ3D-nuScenes within reach of, or above, Let Occ Flow, using no external models. Static-region accuracy must not degrade.

Why not just…?

Why not just mask moving objects and trust the network to generalize? Shared weights do let a network trained only on static pixels predict depth on cars from visual context. However, OccFlowNet's 10.05% dynamic-class gain over naive filtering, and the 17.0% Abs Rel gain in From-Ground-To-Objects, both indicate that context alone leaves a large error. The masking baseline measures this gap directly on our data. The co-moving and stopped-vehicle subsets described below show where context-only generalization breaks.

Why not just use pretrained flow and segmentation, as Let Occ Flow does? This works, and it is the strongest competitor. Its limits are inherited from the external models: out-of-vocabulary objects, boundary bleed, and weak flow for motion along the camera axis, which produces almost no image displacement (Tekin et al.). If our constraint-only model matches it overall, the claim of supervision without external models stands. If it loses overall but wins on axial-motion and co-moving subsets, the honest conclusion is that the two are complementary, and the combined variant tests that.

Why not just distill depth from a large pretrained monocular depth model? This is a real threat. Such models predict plausible shape on cars, but their depth is relative, and the file notes that self-supervised monocular depth cannot recover scale without an extra metric cue. A single per-frame alignment to the static map does not place each object correctly. If the distillation baseline matches ours on dynamic Abs Rel, we will report that. We would then reframe the bounds as the per-object metric anchor for such pseudo-labels, and test that combination, rather than as a replacement for them.

Why not just keep a flat top-down grid with height packed into channels? Occ3D shows BEVFormer at 26.88 mIoU against 28.53 for a full voxel model, and SliceOcc shows height resolution helping. Neither result concerns dynamic objects, however. The representation ablation settles the question here. If the contact term helps equally on a flat grid, height preservation is not needed for this method and we will say so.

Proposed Experiments

We train and evaluate on nuScenes surround video, using LiDAR only for evaluation, and evaluate occupancy on Occ3D-nuScenes. We use SemanticKITTI for RayIoU comparison with Let Occ Flow and SelfOcc, and Cityscapes for monocular comparison with From-Ground-To-Objects. The metrics are Abs Rel overall and on LiDAR points inside annotated dynamic boxes, occupancy IoU and mIoU with dynamic classes reported separately, and RayIoU. Boxes are used only to select evaluation points.

The decisive diagnostic is a "violation rate": the fraction of dynamic-object pixels whose predicted depth exceeds the LiDAR background depth behind them, or whose predicted base sits more than 0.4 m (one Occ3D voxel) above or below the ground. We compute it on two mined subsets. The first contains vehicles with near-zero relative velocity to the ego-vehicle, the infinite-depth case. The second contains vehicles that stop and restart, the map-absorption case.

The method succeeds if it cuts the violation rate sharply relative to masking on both subsets, and if the oracle-map variant shows that most of the remaining error is due to map quality rather than to the bounds themselves. It fails if the self-built map absorbs moving objects often enough that $D_s$ is wrong on foreground pixels. In that case, the measured tightness of the bounds against LiDAR will show it before any downstream metric does.