Hidden, Not Gone: Object Permanence in Dynamic Gaussian Reconstruction

Methods that rebuild a moving 3D scene from video quietly assume that anything they stop seeing has stopped existing. A person who walks behind a pillar for three seconds and walks out again becomes, inside today's dynamic reconstructions, two unrelated people. Their 3D model is deleted or frozen while hidden, and a new one is grown when they reappear. This follows from how these systems were built. Novel view synthesis, meaning the rendering of a scene from camera positions never seen in training, started from a static-world contract: every camera sees the same unchanging 3D content. The dynamic extensions relaxed that contract one frame at a time. Their ways of linking an object across time are velocity, smoothness, and sliding windows, and all of them break when the object vanishes entirely.

We propose to let the reconstruction itself explain the disappearance. If the rebuilt static scene shows a wall between the camera and an object's last known position, then "I can't see it" becomes evidence of a specific kind: the object must lie somewhere in the region that the wall hides from the camera. Call this region the shadow volume. We will keep hidden objects in the model as dormant groups of 3D Gaussians, which are small colored, semi-transparent blobs that together render the scene. Each dormant group is constrained to stay inside its shadow volume while unseen. When a new moving object emerges at the edge of a shadow volume, it is linked back to its earlier identity. The core claim is that this negative evidence ("not visible, therefore behind something") is a strong, cheap, category-agnostic signal. It can carry both identity and geometry across occlusions lasting hundreds of frames, which neither 2D trackers nor current dynamic reconstructions exploit.

What exists and where it stops

Dynamic Gaussian methods link objects across time locally. Split4D gives each Gaussian a velocity and a learned identity feature, and lets 3D trajectories glue identity across frames without a 2D tracker. That reaches 0.893 mIoU on its Multi-Human data against 0.592 for the tracker-dependent SA4D. Once a trajectory is interrupted by full occlusion, though, nothing ties the segment after the gap to the segment before it. SWinGS splits long videos into adaptive windows, each with its own reference shape, so an object hidden across several windows loses its parameterization. R5DGS attaches 16-dimensional identity vectors to Gaussians and supervises them with SAM2+DEVA video masks. Its authors state that this makes it prone to identity switches under severe occlusion. These three papers motivate the gap: identity in 4D reconstruction is either local or inherited from a 2D tracker that fails exactly when occlusion is long.

Two papers come closest and also supply method. USplat4D models per-Gaussian uncertainty stretched along the camera's viewing ray and propagates motion from well-observed anchor Gaussians to partly occluded ones. We reuse that anisotropic uncertainty for dormant objects. USplat4D still needs an anchor that is visible, and its authors note that if the underlying tracker loses an object entirely, no anchors remain. Guess The Unseen reconstructs humans through long occlusions by filling unseen parts with a diffusion prior. It reports 23.60 PSNR on Panoptic, but it depends on the SMPL human body model and does not reason about where a hidden person can be.

On the tracking side, PLANET grounds tracking queries in monocular depth and keeps a two-resolution memory reaching about 200 frames. QueST lifts tracks to 3D and constrains them to plausible motions. Both keep sparse points or box centers rather than renderable geometry, and neither uses scene occluders to restrict where a hidden target may be. QueST reports failure beyond roughly 30 frames at more than 80% occlusion. 4RC attends over the whole video in a single feed-forward pass and can bridge brief disappearances. It has no discrete object identities.

The distractor-removal line also constrains our design. RobustSplat shows that moving objects cause loss spikes, and that these spikes make 3D Gaussian Splatting spawn "floater" Gaussians which scar the background. Any method that keeps hidden objects must therefore stop dormant Gaussians from being densified or pruned on photometric evidence they cannot receive.

Core Mechanism

The representation splits the scene into static Gaussians and dynamic object groups, each carrying an identity feature in the style of Split4D. Each group also has a discrete state: active, occluded, or exited.

At every frame we render the group's predicted position and compare its depth with the rendered depth of the static scene and of other objects. There are three cases. If the group projects inside the image and something nearer covers it, the group is occluded. If it projects outside the image, it has exited. Otherwise it is active, which means it should be visible, and unexplained invisibility counts as a tracking failure.

Occluded groups are exempt from photometric loss, opacity decay, and densification, which avoids the RobustSplat failure. Their motion is governed by an uncertainty-growing prior together with a hiding constraint. Let $x_i$ be the center of a Gaussian $i$ in a dormant group, $\pi(x_i)$ its pixel projection, $z(x_i)$ its depth, $o_i$ its opacity, and $D_t(u)$ the rendered depth of all other content at pixel $u$ and time $t$. We penalize

$$ \mathcal{L}_{\text{hide}} = \sum_i o_i \,\mathbb{1}[\pi(x_i)\in\Omega]\,\max\big(0,\; D_t(\pi(x_i)) + \epsilon - z(x_i)\big), $$

where $\Omega$ is the image domain and $\epsilon$ a small margin. The penalty is zero only if every dormant Gaussian lies behind whatever the camera currently sees, that is, inside the shadow volume. A static camera behind a narrow pillar yields a small shadow volume and a tightly constrained hidden trajectory. A wide occluder yields a loose one, and in that case the anisotropic uncertainty from USplat4D grows accordingly.

Re-association happens when a new dynamic group is spawned. We score its match against each dormant group using two factors. The first is reachability: can the dormant group get from its entry point to the new emergence point within the elapsed time while staying inside the shadow volume? The second is render agreement: does the dormant group's Gaussians, posed at the new location, explain the new observations photometrically? Reachability is the geometric part that 2D re-identification lacks. It should matter most when targets look alike, as on DanceTrack, where appearance-based trackers fall from about 60–70 HOTA on MOT17 to about 40–50. Accepted matches merge the two groups. The hidden trajectory is then refit as a path that is smooth, stays inside the shadow volume, and connects both ends, so the scene can be rendered from new viewpoints during the gap.

Research Program

We will build the method on top of a Split4D-style representation for monocular and sparse multi-view video. We will then study three questions. The first is how much identity accuracy the hiding constraint and reachability add over keeping hidden objects and matching on appearance alone. The second is whether constraining the hidden trajectory improves rendering accuracy of the hidden object from other cameras during the gap. The third is where the approach breaks: occluders that are themselves moving, wide occluders that leave the trajectory nearly unconstrained, and poor static depth. Because the occlusion test depends on the quality of the static reconstruction, we will report results stratified by background depth error.

Baselines

The naive reference point is a Split4D-style reconstruction with no permanence mechanism, where reappearing objects simply become new identities. It shows how often the problem occurs and how much it costs. Split4D's 0.893 mIoU on Multi-Human is the published figure for this family, but it was measured on clips without long full occlusions.

The strongest existing competitor for identity is tracker-supervised reconstruction in the style of R5DGS. It uses SAM2+DEVA masks to supervise Gaussian identity, and SAM2 carries its own long-range memory. We run it on the same videos and the same optimization schedule, and with the same Gaussian budget and iteration count as our method. Beating it means higher post-reappearance identity accuracy and IDF1 on our occlusion splits at matched rendering quality. No reported result found on Lacuna for R5DGS on long-occlusion splits.

For hidden geometry, the strongest competitor is USplat4D, whose reported DyCheck PSNR is 19.63 against 19.32 for MoSca. We expect to at least match it on DyCheck, which has few full occlusions, and to beat it in rendering the hidden interval on Panoptic held-out views.

For human scenes we also compare against Guess The Unseen, which reports 23.60 PSNR on Panoptic. That number was measured under its own protocol, so we rerun it on our splits.

For the feed-forward route we run 4RC and read identities from its displacement queries.

The key ablations remove one ingredient each. Removing $\mathcal{L}_{\text{hide}}$ keeps dormant groups but moves them with a constant-velocity prior. Removing reachability from the re-association score leaves appearance-only matching. Together these isolate whether negative evidence, rather than mere retention, does the work.

Why not just…?

Why not just feed SAM2+DEVA identities into a dynamic Gaussian model? This is the most serious threat, because SAM2's memory may bridge many occlusions on its own. The evidence against relying on it is R5DGS's own reported identity switches under severe occlusion, plus Split4D's finding that tracker-dependent lifting reaches 0.592 versus 0.893 mIoU. Neither result isolates long full occlusion, so our splits settle the question directly. If SAM2-supervised identities match ours on reappearance accuracy, the identity claim fails. The remaining contribution would then be constrained hidden geometry alone, and we would report it that way.

Why not just keep hidden objects and merge them afterwards using appearance embeddings? This is exactly our ablation without reachability and without $\mathcal{L}_{\text{hide}}$. DanceTrack's appearance collapse suggests it should fail when targets look alike. The experiment on look-alike crossover sequences will show whether reachability adds anything beyond appearance.

Why not just interpolate the gap with a spline or a Kalman filter? DanceTrack documents strongly nonlinear crossovers that defeat constant-velocity models. USplat4D shows that monocular uncertainty along the viewing ray makes unconstrained smoothing drift into floaters. Our constant-velocity ablation measures this directly through held-out-view error during the gap.

Why not just let a global-attention model like 4RC see the whole video? 4RC has no discrete identities, and its authors note difficulty with chaotic motion. Running it on the same splits will show whether global context alone recovers identity.

Proposed Experiments

No existing benchmark pairs dense 4D reconstruction with long disappear-and-reappear events, so we build two evaluation splits from existing data.

The first uses Panoptic multi-camera captures. We choose one input camera per sequence and keep the segments in which a person is fully hidden from that camera for at least 60 frames by another person or by furniture. The other cameras provide ground truth for rendering the hidden person, which we score with PSNR and LPIPS, a learned perceptual distance, on held-out views during the gap. They also provide ground truth for identity after reappearance.

The second uses PointOdyssey, which has visibility labels and sequences of thousands of frames. We select tracks that are invisible for long stretches and measure post-reappearance 3D trajectory error, identity re-association accuracy, and survival rate. The reference there is PIPs++ at 50.47% survival on the full benchmark.

As 2D sanity checks we include DyCheck, to confirm that rendering does not regress, and a static-camera DanceTrack subset scored by projecting rendered identity masks and computing HOTA and IDF1. PLANET's 72.1 HOTA and 78.0 IDF1 on the full benchmark serve as context, not a direct target.

The decisive check is the ablation gap on look-alike crossovers. If the full model beats both the appearance-only merge and SAM2-supervised identities on reappearance accuracy, and the gap widens as occluders narrow and constrain the shadow volume more tightly, then negative evidence is doing the work. If the gap does not track occluder width, the mechanism is not what we claim.