Remembering What Moving Objects Hide: Motion-Gated Scene Memory for Camera-Based Scene Completion

A self-driving car's cameras see only visible surfaces. The sidewalk behind a passing truck, or the curb behind a cyclist, is missing from the current frame. It was often visible a second or two earlier, before the object moved in front of it. Semantic scene completion is the task of predicting, for every cell of a 3D grid around the vehicle, whether it is occupied and by what class, including cells the camera cannot currently see. Camera-based systems for this task should be able to remember that sidewalk. They cannot. Stacking past frames corrupts the prediction, because moving objects leave smeared "ghost" traces. Aligning past frames with optical flow, the per-pixel motion field between images, accumulates drift. CurriFlow warps historical features by optical flow and reports that accuracy peaks with just one past frame, then degrades as more are added. The field has in effect given up on long memory. Some of the strongest recent systems, such as DISC, are single-frame models that beat multi-frame ones.

We propose to separate what should be remembered from what should not, before anything is fused. Static structure (road, buildings, parked cars) can be accumulated over long horizons in a world-fixed 3D memory using only the vehicle's own pose, with no flow and therefore no flow drift. Moving objects are kept out of that memory and handled from the current frame alone. The deciding signal is a comparison between two depth estimates for each pixel. One is the depth a single image implies, from a monocular depth network that uses semantic context to fix absolute scale. The other is the depth the static memory predicts at that pixel given the car's motion. Where the two disagree, something has moved. The central testable claim is that with this gate in place, completion accuracy rises steadily with history length instead of peaking at two frames. The gain should be concentrated on voxels (3D grid cells) that are currently hidden behind moving objects.

Core Claim

Long-horizon temporal fusion fails in camera-based scene completion because static and dynamic content are fused through a single alignment mechanism. For static content, that mechanism is unnecessary. Its errors on dynamic content then contaminate everything. A per-pixel motion gate built from monocular metric depth lets static observations enter a persistent world-frame memory with no dependence on flow, while dynamic pixels are excluded. This turns the moving occluder from a source of ghosting into a source of free completion. The background it hides now was observed before, and the memory holds it.

The gate is simple enough to write down. Let $D_t(u)$ be the monocular metric depth at pixel $u$ in frame $t$. Let $\hat D_t(u)$ be the depth obtained by rendering the static memory into the current camera using ego-pose. Let $c(u)$ be the predicted semantic class at that pixel. The probability that the pixel is dynamic is

$$p_{\text{dyn}}(u) = \sigma\!\left(\alpha \,\frac{\max(0,\ \hat D_t(u) - D_t(u))}{D_t(u)} + \beta_{c(u)}\right),$$

where $\sigma$ is the logistic function, $\alpha$ is a learned sensitivity, and $\beta_c$ is a learned per-class bias. The bias encodes that cars are often moving and buildings never are. The residual is one-sided for a reason: a new object appearing in front of remembered background makes observed depth shorter than expected. Longer-than-expected depth usually means disocclusion, which the memory should accept. Pixels are written to memory with weight $1 - p_{\text{dyn}}(u)$, and memory cells that are repeatedly contradicted decay. Because the class bias is only a prior, a parked car that stays consistent with memory is admitted as static. Class-based masking would discard it.

Relation to Prior Work

Two cross-direction systems motivate the design and define the gap. CDScene observes that multi-view matching fails on moving objects. It routes dynamic structure through a monocular depth branch and static structure through a short-window stereo/temporal branch, then fuses them in voxel space with attention. We borrow its insight that monocular depth is immune to motion. CDScene uses monocular depth as a parallel feature source over consecutive frames. We use it as a consistency test that controls what a long-lived memory is allowed to store. CurriFlow supplies the key negative result: optical-flow warping degrades beyond two frames. That result constrains our design to avoid flow on the static path entirely.

The single-frame systems Ocean, DISC and DepthSSC serve three roles in this proposal:

  • Strongest competitors. They improve instance separation and alignment within one frame, using instance masks from a segmentation model, class-decoupled queries, and learned per-voxel 2D-to-3D alignment respectively.
  • Base model. We reuse one of them as the per-frame network our memory attaches to.
  • Conceptual contrast. Behind a moving occluder, any single-frame model can only guess. Our memory can recall.

The context-aware depth literature supplies the gate's main input. ScaleDepth splits metric depth into a semantically predicted scene scale and a relative depth map. Depth Anything aligns depth features with a semantic encoder so that surface texture is not mistaken for geometry. Their accuracy also sets the gate's noise floor. ScaleDepth reports an absolute relative error of 0.048 on KITTI. We therefore expect the gate to miss objects whose motion changes depth by less than roughly 5 to 10 percent along the viewing ray, such as a pedestrian walking slowly toward the camera. The class prior and memory decay must cover that regime.

Work on distractor-free reconstruction frames the conceptual gap. RobustSplat and 3DGS-HPC mask transient objects out of static reconstructions. Split4D separates dynamic from static Gaussians, the small 3D blobs that make up a splatted scene. All three leave the background behind removed objects hollow; Split4D names this explicitly. SGC-Net completes urban surfaces behind parked vehicles, but only in static 8 m chunks, from geometry alone. We complete behind moving occluders by recall where history exists, and fall back on learned completion where it does not.

Research Program

The system has three parts attached to a per-frame completion network.

The first part is a monocular metric depth module, frozen or lightly fine-tuned, of the ScaleDepth or Depth Anything type. It produces $D_t$ and pixel semantics.

The second part is a sparse, hash-indexed voxel memory in world coordinates at the benchmark's 0.2 m resolution. Each cell stores a feature vector, an occupancy logit, and a visibility count. Storing only observed cells avoids the cubic memory cost of dense grids that CDScene and SGC-Net inherit, and it is what makes horizons of dozens of frames affordable. Each frame, the memory is rendered into the current camera to produce $\hat D_t$. The gate is computed, and static-weighted features are written back using ego-pose from the dataset's odometry.

The third part is a fusion step. The current-frame network queries memory cells inside its output volume and fuses them with its own features through cross-attention. Dynamic regions receive no memory features, only current-frame instance reasoning.

Training uses the standard scene-completion labels plus an auxiliary loss on the gate. The gate target comes from LiDAR segmentation labels that distinguish moving from static instances, which SemanticKITTI provides (SemanticKITTI); the camera-only system never sees LiDAR at test time. A second analysis thread asks where the gains come from. We split evaluation voxels into three groups: currently visible; currently occluded by static structure; and currently occluded by a moving object but visible within the history window. The third group is computed from ground-truth poses, LiDAR moving labels, and ray casting. We expect memory to help the third group most, the second somewhat, and the first little. We will also check whether the benchmark's ground-truth voxels themselves contain traces of moving objects, and report the effect on the third group if they do.

Baselines

The naive reference is the chosen per-frame network run on the current frame only. Its gap to our system on the moving-occluder voxels shows what single-frame reasoning can and cannot guess.

The strongest existing single-frame methods are Ocean and DISC:

Method SemanticKITTI mIoU SSCBench-KITTI-360 mIoU
Ocean 17.40 20.28
DISC (SemanticKITTI hidden test) 17.35 20.55

We will attach our memory to one of these two (the one we can reproduce most faithfully). That way the comparison isolates the memory from the base architecture: same backbone, same training schedule, same image resolution. We will count it as beating them if it improves overall mIoU on both benchmarks. The larger requirement is a clear improvement on the moving-occluder voxel group.

The strongest multi-frame competitors are CurriFlow (16.9 mIoU on the SemanticKITTI validation set) and CDScene (17.24 mIoU on SemanticKITTI and 19.04 on SSCBench-KITTI360). For these we will rerun both CurriFlow-style flow warping and our memory at matched history lengths of 1, 2, 4, 8 and 16 frames, on the same base network. This reproduces the known peak at two frames and tests whether ours keeps rising.

Two ablations isolate the key ingredient. The first removes the gate and writes every pixel to memory; this measures ghosting. The second replaces the gate with a pure semantic mask that excludes all vehicle and person classes; this tests whether the depth-consistency residual adds anything beyond class labels.

All variants will be compared at equal training compute and equal tuning budget. Inference latency will be reported, since CurriFlow runs at 2.24 s per frame on an RTX 3090.

Why not just…?

Why not just stack more frames with ego-pose alignment and let attention sort it out? This is the ungated ablation. The evidence against it is the ghosting that CDScene documents when moving objects violate cross-frame consistency. We will measure it directly: if ungated memory matches gated memory on the moving-occluder voxels and on dynamic classes, the gate is unnecessary and we will say so.

Why not just mask out every car and pedestrian by class? This is a real threat, because semantic segmentation is mature and the mask is nearly free. Its weakness is structural. Parked cars dominate KITTI streets, and SGC-Net exists precisely because they are major occluders. A class mask throws away their stable geometry and cannot tell a parked car from a moving one. The class-only ablation settles the question. If it matches the depth-residual gate on both the static-car and dynamic-class metrics, the simpler version wins and becomes the recommended method.

Why not just use a stronger single-frame model, since DISC already beats multi-frame methods? A single-frame model has no information about background currently hidden behind a moving truck; it can only produce a plausible guess from priors. This matters most for fine structure: SGC-Net's gains on 15 cm curbs illustrate how much detail pure priors must otherwise invent. If our memory attached to DISC-class networks gives no gain on the moving-occluder group, the premise is wrong.

Why not just use better scene flow instead of optical flow? Better motion estimation would help the dynamic path. Our argument is that static content needs no motion estimate beyond ego-pose, so any flow model adds error there, of the kind CurriFlow reports. The history-length sweep compares flow-based fusion and gated memory head to head.

Decisive Checks

The experiments run on SemanticKITTI semantic scene completion (validation set for the sweeps, hidden test set for the final number), on SSCBench-KITTI-360, and on SemanticKITTI-C. On SemanticKITTI-C, CDScene reports 11.80 mean mIoU, and the gate's reliance on monocular depth under fog and snow is its likeliest failure. Metrics are IoU and mIoU, overall, per dynamic class, and on the three visibility groups.

The proposal succeeds if two things hold. First, gated-memory mIoU keeps increasing up to at least eight frames, whereas the flow-warped equivalent peaks at two. Second, the gain over the matched single-frame base is several times larger on moving-occluder voxels than on visible voxels. It fails outright if the class-only mask matches it, or if the gains vanish under SemanticKITTI-C corruption. In that second case the depth residual would be too noisy to serve as a gate in bad weather, which would itself be a useful negative result about depth-based motion reasoning.