TL;DR
ReCan-GS is a 4D Gaussian Splatting framework that automatically separates stable scene motion from transient "distractors" like passing people, shadows, or smoke. By deriving a novel uncertainty metric directly from the math of the learned deformation field, it produces clean dynamic reconstructions without needing pre-trained segmentation models or manual masks.
What Problem Does This Paper Address?
Reconstructing dynamic 3D scenes from a single moving camera is a "holy grail" in computer vision, but real-world video is messy. Most videos contain distractors: transient occluders (like a person walking in front of your subject), moving shadows, reflections, or volumetric effects like smoke and fog.
Current methods usually fall into two camps, both of which struggle here:
- Static distractor-removal: These assume the background is still and treat any motion as noise. If your target object is moving (e.g., a car driving), these methods accidentally delete it.
- Standard dynamic reconstruction (like 4DGS): These try to model every single pixel they see. When a transient occluder appears, the model tries to force it into the 3D representation, resulting in "ghosting" artifacts—blurred, semi-transparent remnants of the distractor that haunt the final render.
ReCan-GS aims to "reconstruct what you can"—preserving valid, consistent subject motion while routing inconsistent "unreconstructable" observations into a separate layer.
Methods
The core innovation of ReCan-GS is how it identifies distractors. Instead of using a pre-trained AI to recognize "people" or "cars," it looks at deformation-space instability.
The Two-Layer Architecture
The system models the scene using two distinct sets of 3D Gaussians:
- Core Gaussians: These represent the "true" scene (static background + dynamic subject). They live in a "canonical" space and are moved over time by a learned deformation field (an MLP).
- Distractor Gaussians: These are per-view layers designed to soak up transient, view-specific "junk" that doesn't fit the core model.
The Jacobian Uncertainty Score
To decide which Gaussian belongs where, the authors compute a Jacobian-based uncertainty score. They look at the gradients of the deformation field—essentially asking, "How sensitive is the predicted motion to tiny changes in position or time?"
If a region has high sensitivity (high variance in these gradients), it suggests the model is struggling to find a consistent geometric explanation for what it's seeing. This "Jacobian uncertainty" acts as a proxy for reconstruction ambiguity.
The authors propose two ways to use this score:
- Loss-based Regularization: During training, a loss term penalizes Gaussians that are both opaque and uncertain, effectively "pushing" distractors into the per-view distractor layer.
- Post-Hoc Pruning: After training, the system can simply delete the top-X% most uncertain Gaussians to clean up residual artifacts.
Datasets
The authors evaluated ReCan-GS across three distinct regimes:
- Synthetic Dynamic Scenes: Bouncing Balls (occluded by a runner) and Squatting (subject partially hidden by a figure).
- Real-World Dynamic Scenes: Two iPhone captures—Jeep (stop-motion with hands moving the toy and casting shadows) and Bottle (frequently occluded by a walking person).
- Smoke Scenes: Four synthetic scenes from the SmokeSeer dataset, which pair unstable smoke observations with smoke-free ground truth.
- Static Benchmark: The NeRF On-the-go benchmark to test how well the method generalizes to traditional static distractor removal.
Key Insights and Ideas
- Geometry over Appearance: Unlike prior work that identifies distractors based on color error or semantic labels, ReCan-GS uses the stability of the motion field. This allows it to handle distractors that look like the background but move inconsistently.
- Reconstructability vs. Speed: A common misconception is that distractors are just "fast-moving things." The authors show that as you add more camera views (better coverage), the uncertainty score for a valid moving object drops, even if it's fast. This proves the metric captures ambiguity, not just velocity.
- The "Smoke" Aha-Moment: Smoke is notoriously hard because it's semi-transparent and photometrically stable but geometrically chaotic. ReCan-GS treats smoke as a high-uncertainty distractor and prunes it effectively without needing a specialized "smoke model."
Key Results
- Distractor Suppression: In the Jeep scene, human judges and Vision-Language Models (VLMs) like GPT-5 rated ReCan-GS as having near-zero distractor retention. In contrast, standard dynamic methods (DeformableGS) showed distractors in over 70% of frames.
- Quantitative Gains: On the Bouncing Balls scene, ReCan-GS achieved 29.4 dB PSNR, outperforming all baselines. On smoke removal, it saw gains of up to 6.2 dB over the strongest general distractor-removal baselines.
- Qualitative Quality: As seen in Figure 3, while other methods leave "ghosts" or blurry trails where a hand or person was, ReCan-GS recovers a sharp, clean subject.
Limitations and Future Work
- Static Benchmarks: ReCan-GS is slightly less effective than specialized static-scene methods on standard static benchmarks (like NeRF On-the-go). The authors aren't entirely sure why yet, but it suggests the method is currently best suited for truly dynamic scenes.
- Ambiguity: If a valid part of the subject is very poorly observed (e.g., seen from only one angle for a split second), the system might mistakenly label it as uncertain and prune it.
- Future Directions: The authors suggest that combining their geometric uncertainty with semantic priors (like DINO features) could create an even more robust system that understands what it is looking at while verifying how well it can be reconstructed.
Summary updated