Reconstruct What You Can: Distractor-free 4D Gaussian Splatting

Joel Julin, Liuyue Xie, Ananya Bal, Mose Sakashita, Koichiro Niinuma, Laszlo A. JeniarXiv 2026 · arXiv

TL;DR

ReCan-GS is a 4D Gaussian Splatting framework that automatically separates stable scene motion from transient "distractors" like passing people, shadows, or smoke. By deriving a novel uncertainty metric directly from the math of the learned deformation field, it produces clean dynamic reconstructions without needing pre-trained segmentation models or manual masks.

What Problem Does This Paper Address?

Reconstructing dynamic 3D scenes from a single moving camera is a "holy grail" in computer vision, but real-world video is messy. Most videos contain distractors: transient occluders (like a person walking in front of your subject), moving shadows, reflections, or volumetric effects like smoke and fog.

Current methods usually fall into two camps, both of which struggle here:

  1. Static distractor-removal: These assume the background is still and treat any motion as noise. If your target object is moving (e.g., a car driving), these methods accidentally delete it.
  2. Standard dynamic reconstruction (like 4DGS): These try to model every single pixel they see. When a transient occluder appears, the model tries to force it into the 3D representation, resulting in "ghosting" artifacts—blurred, semi-transparent remnants of the distractor that haunt the final render.

ReCan-GS aims to "reconstruct what you can"—preserving valid, consistent subject motion while routing inconsistent "unreconstructable" observations into a separate layer.

Figure 1. From monocular frames of noisy, dynamic scenes (a), our method decomposes reconstructable (green) and unreconstructable motion (red) (b), enabling distractor-free dynamic reconstruction (c).

Methods

The core innovation of ReCan-GS is how it identifies distractors. Instead of using a pre-trained AI to recognize "people" or "cars," it looks at deformation-space instability.

The Two-Layer Architecture

The system models the scene using two distinct sets of 3D Gaussians:

  • Core Gaussians: These represent the "true" scene (static background + dynamic subject). They live in a "canonical" space and are moved over time by a learned deformation field (an MLP).
  • Distractor Gaussians: These are per-view layers designed to soak up transient, view-specific "junk" that doesn't fit the core model.

The Jacobian Uncertainty Score

To decide which Gaussian belongs where, the authors compute a Jacobian-based uncertainty score. They look at the gradients of the deformation field—essentially asking, "How sensitive is the predicted motion to tiny changes in position or time?"

If a region has high sensitivity (high variance in these gradients), it suggests the model is struggling to find a consistent geometric explanation for what it's seeing. This "Jacobian uncertainty" acts as a proxy for reconstruction ambiguity.

Figure 2. An overview of the ReCan-GS pipeline, showing how the Jacobian-based uncertainty signal guides the optimization to penalize unstable motion in the core layer.

The authors propose two ways to use this score:

  1. Loss-based Regularization: During training, a loss term penalizes Gaussians that are both opaque and uncertain, effectively "pushing" distractors into the per-view distractor layer.
  2. Post-Hoc Pruning: After training, the system can simply delete the top-X% most uncertain Gaussians to clean up residual artifacts.

Datasets

The authors evaluated ReCan-GS across three distinct regimes:

  • Synthetic Dynamic Scenes: Bouncing Balls (occluded by a runner) and Squatting (subject partially hidden by a figure).
  • Real-World Dynamic Scenes: Two iPhone captures—Jeep (stop-motion with hands moving the toy and casting shadows) and Bottle (frequently occluded by a walking person).
  • Smoke Scenes: Four synthetic scenes from the SmokeSeer dataset, which pair unstable smoke observations with smoke-free ground truth.
  • Static Benchmark: The NeRF On-the-go benchmark to test how well the method generalizes to traditional static distractor removal.

Key Insights and Ideas

  • Geometry over Appearance: Unlike prior work that identifies distractors based on color error or semantic labels, ReCan-GS uses the stability of the motion field. This allows it to handle distractors that look like the background but move inconsistently.
  • Reconstructability vs. Speed: A common misconception is that distractors are just "fast-moving things." The authors show that as you add more camera views (better coverage), the uncertainty score for a valid moving object drops, even if it's fast. This proves the metric captures ambiguity, not just velocity.
  • The "Smoke" Aha-Moment: Smoke is notoriously hard because it's semi-transparent and photometrically stable but geometrically chaotic. ReCan-GS treats smoke as a high-uncertainty distractor and prunes it effectively without needing a specialized "smoke model."
Figure 4. Smoke removal results. ReCan-GS detects deformation-space instability to enable clean smoke-free reconstructions.

Key Results

  • Distractor Suppression: In the Jeep scene, human judges and Vision-Language Models (VLMs) like GPT-5 rated ReCan-GS as having near-zero distractor retention. In contrast, standard dynamic methods (DeformableGS) showed distractors in over 70% of frames.
  • Quantitative Gains: On the Bouncing Balls scene, ReCan-GS achieved 29.4 dB PSNR, outperforming all baselines. On smoke removal, it saw gains of up to 6.2 dB over the strongest general distractor-removal baselines.
  • Qualitative Quality: As seen in Figure 3, while other methods leave "ghosts" or blurry trails where a hand or person was, ReCan-GS recovers a sharp, clean subject.
Figure 3. Qualitative results. ReCan-GS recovers a clean, stable core reconstruction while dynamic baselines produce ghosting.

Limitations and Future Work

  • Static Benchmarks: ReCan-GS is slightly less effective than specialized static-scene methods on standard static benchmarks (like NeRF On-the-go). The authors aren't entirely sure why yet, but it suggests the method is currently best suited for truly dynamic scenes.
  • Ambiguity: If a valid part of the subject is very poorly observed (e.g., seen from only one angle for a split second), the system might mistakenly label it as uncertain and prune it.
  • Future Directions: The authors suggest that combining their geometric uncertainty with semantic priors (like DINO features) could create an even more robust system that understands what it is looking at while verifying how well it can be reconstructed.

Summary updated

Abstract

Dynamic reconstruction from casual video must preserve valid subject motion while suppressing transient occluders, shadows, and volumetric effects. Existing methods either suppress valid motion through static-scene assumptions or model all observations jointly, producing ghosting and unstable geometry. We introduce ReCan-GS, a 4D Gaussian Splatting framework that separates coherent scene content from inconsistent observations. It derives a Jacobian-based deformation-sensitivity score from the learned deformation field, using it as an empirical proxy for reconstruction ambiguity. The score guides separation into dynamic Core Gaussians and per-view Distractor Gaussians, or post-hoc pruning of sensitive Core Gaussians. The same criterion supports removal of spatial occluders, photometric artifacts, and evolving smoke or fog without masks, pretrained segmenters, or medium-specific supervision. Experiments on synthetic and real dynamic scenes and synthetic smoke/fog scenes demonstrate its usefulness across the evaluated distractor regimes.
BibTeX
@misc{julin2026reconstruct,
  author = {Joel Julin and Liuyue Xie and Ananya Bal and Mose Sakashita and Koichiro Niinuma and Laszlo A. Jeni},
  title = {{Reconstruct What You Can: Distractor-free 4D Gaussian Splatting}},
  year = {2026},
  eprint = {2611.16259},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16259},
}
Figure