When you insert a new object into a 3D scene, it should cast shadows on the floor, dim the wall behind it, and pick up the room's light on its own surface. Current methods get these effects wrong in two ways. Image-based diffusion models paint plausible shadows one view at a time, so the shadow on the floor can point a different way in every camera. 3D methods keep the background frozen, so the object looks like a hologram. Learning these effects properly needs paired data: the same scene photographed with and without the object. ObjectDrop collected such pairs by hand, 2,500 tripod shots in all. Our observation is that a multi-view video of a dynamic scene already holds thousands of these pairs. Each time a person walks across a room, every floor and wall patch is seen both shadowed and unshadowed, under the same lights, from several calibrated cameras.
We propose to exploit the very thing reconstruction pipelines treat as a nuisance, the breaking of the static-scene assumption. First we split a captured 4D scene into a static background and its native movers. Then we fit a compact, explicitly 3D model of the scene's light from how the movers darken the background and how their own brightness changes as they move. Finally we reuse that light model to insert new moving objects whose shadows and shading agree with the scene's true lights, consistently across every view and every timestep. In plain terms, the people already in the video calibrate the lighting, and the inserted object inherits it.
Core Claim
In multi-view dynamic captures, the native moving objects supply enough "with and without" supervision to recover a scene-specific model of light occlusion and incident light. That model is cheap to fit and, because it lives in 3D, is consistent across views by construction. Our hypothesis is that inserted moving objects rendered through this model will match real shadow pixels better than two alternatives: per-view diffusion harmonization and single-image light estimation. We test this on a protocol where ground truth exists, namely re-inserting a held-out real mover whose actual shadows were never used for fitting.
The roles of the four directions are uneven. Dynamic scene reconstruction supplies the data source. Global illumination consistency is the target. The Janus problem appears in a specific, under-noticed form we call lighting Janus: per-view generative shadows that contradict each other across cameras, which is the shading analogue of the multi-faced artifacts described in DreamBooth3D. Cross-modal data scarcity is the framing lens. Paired object/effect data is a scarce paired modality in the same way synchronized audio-visual data is, and we answer it the way Sonic4D and VividVoice answer audio scarcity. Sonic4D does so through physics-structured modules rather than end-to-end paired training. VividVoice does so by building its own pairs. We do the same, except our pairs come from real capture rather than synthesis.
Relation to Prior Work
Three insertion works motivate the gap. InseRF edits a single reference view and lifts it to 3D, which avoids multi-view contradictions, but its own analysis says inserted objects cast no shadows and behave like holograms. D3DR harmonizes an inserted object's colors into a static Gaussian scene using a personalized diffusion model and IC-Light variants. It adapts the object to the scene but does not alter the background, and it assumes a static scene. ObjectDrop learns the full object-to-scene effect from counterfactual pairs, but it works in 2D and the authors note its shadows can conflict with the scene's real light direction. We take ObjectDrop's central idea, that counterfactual pairs teach light transport. We differ in mining the pairs from dynamic capture and fitting a 3D light model instead of a 2D image prior.
Two dynamic-scene works supply the method. Split4D groups Gaussians (small 3D blobs that together render the scene) into static and moving parts by their trajectories, without 2D trackers. It reports 0.893 mIoU on Multi-Human versus 0.592 for SA4D. It also states that removing or adding movers leaves shadow artifacts because illumination is unmodeled. That limitation is our entry point, and its decomposition is our first stage. Instruct-4DGS separates a canonical scene from a deformation field and edits only the canonical appearance. We borrow that canonical-plus-motion structure for the inserted object, but we change the background's per-time shading, which Instruct-4DGS never touches.
Carve3D constrains our evaluation design. It measures multi-view consistency by reconstructing a 3D model from generated views and scoring the re-rendering error, a quantity it calls MRC. With that reward it improves MRC from 0.0892 to 0.0606 on the 415-prompt DreamFusion set. We reuse the idea as a diagnostic applied to shadow layers instead of whole objects, to quantify lighting Janus. The inverse-rendering line (Huang et al., Liang et al., and the R3DG baseline in D3DR) is a negative constraint. Full material decomposition is slow and brittle, so we deliberately estimate only lights and occlusion, never surface materials.
Research Program
Stage one: mining counterfactuals. We decompose the capture with Split4D. For each static background Gaussian $i$ we gather its observed color $c_i(t)$ over time, together with the configuration of moving Gaussians at each time $t$.
Stage two: fitting the light model. We model a background patch's color as a static color dimmed by occlusion from movers:
$$c_i(t) = \rho_i \Big( E_0 + \sum_{k=1}^{K} V_{ik}(t)\, L_k\, \max(0, n_i \cdot \omega_k) \Big).$$
Each symbol is defined as follows:
- $\rho_i$ is the patch's effective color.
- $E_0$ is a low-frequency ambient term, represented with spherical harmonics.
- $L_k$ and $\omega_k$ are the intensity and direction or position of $K$ dominant lights.
- $n_i$ is the patch normal.
- $V_{ik}(t) \in [0,1]$ is the soft visibility of light $k$ from patch $i$ at time $t$.
The visibility $V_{ik}(t)$ is computed by splatting the moving Gaussians toward light $k$, a differentiable Gaussian shadow map. Because the static scene and the camera views are fixed, the only things that change over time are the $V_{ik}(t)$. The light parameters are therefore identified from the movers' shadows alone. We also fit the opposite direction, the scene lighting the object. The movers' own shading, computed with the same lights and their occlusion by static geometry, constrains the same $L_k$ and $E_0$. This includes, for instance, the brightening of a person near a window. A residual low-rank color term captures indirect light bounced from movers onto nearby surfaces.
Stage three: insertion. A new object is generated from a single reference view and lifted to a canonical 3D model with a multi-view model, InseRF-style. A deformation field, Instruct-4DGS-style, then animates it along a user-specified trajectory. Its Gaussians enter the shadow-map term as extra occluders, and its own colors are re-shaded with the fitted lights. No diffusion model touches the background. Shadows are therefore a deterministic function of 3D geometry, which rules out lighting Janus by construction. As an optional final step, a light diffusion refinement can add texture detail, and we report results with and without it.
Stage four: guarding against temporal Janus. The inserted object can still mutate over time, for example by growing a second face as it turns. To catch this we compute a time-extended MRC: we rebuild the object from frames at different times, map it back into canonical space, and score the re-rendering error. This step is a diagnostic and an optional regularizer, not the main contribution.
Baselines
The naive reference point is compositing the inserted object into the dynamic scene with no lighting interaction: the background stays frozen and the object keeps its generated colors. This is effectively InseRF behavior extended over time. Every shadow-region metric should improve over it. If we fail to beat it inside shadow masks, the light model is not working.
The strongest existing competitor is per-frame, per-view 2D counterfactual insertion. We render the plain composite in each view and frame and pass it through an ObjectDrop-style insertion model to add shadows. We then optionally re-fit the 4D Gaussians to the edited frames, which averages the views. ObjectDrop reports 23.153 PSNR for removal on its own evaluation set and an 88.9% user preference over AnyDoor for insertion. It is the best documented learner of object-to-scene effects in the file. For video or dynamic-scene shadow accuracy, no reported result was found on Lacuna. Because ObjectDrop's model is not stated to be public, we will use the strongest available open 2D insertion model trained on counterfactual-style data and name it explicitly. It receives identical input frames, masks, and object renders. We tune its guidance strength on the same validation split we use for our method's single hyperparameter, the number of lights $K$.
Our second strong baseline is D3DR's harmonization applied per time slice of the 4D scene. It reports a 0.5 dB PSNR and 0.15 SSIM gain over TIP-Editor and R3DG on its static benchmark. It is the strongest 3D-consistent harmonization method, but it changes only the object. We run it at its published settings on the same canonical object.
The third baseline is R3DG-style inverse rendering run on the static background, with light estimated from static cues only and shadows ray-traced from the inserted Gaussians. It gets the same GPU budget as our method.
Two ablations isolate the key ingredient. The first fits our light model only from the scene's existing static shadows and a single-image light estimate, with the movers removed. It tests whether the movers' counterfactuals are what make the difference. The second keeps everything but replaces the 3D shadow map with a 2D learned shadow predictor applied per view. It tests whether 3D structure is what eliminates lighting Janus.
Beating the field means two things on held-out-mover re-insertion. Shadow-mask PSNR and LPIPS must exceed both the 2D-insertion baseline and the static-cue ablation, with confidence intervals across scenes. Shadow-layer MRC must also be clearly lower than the 2D baseline's.
Why not just…?
Why not just run 2D counterfactual insertion on every frame and let 4D reconstruction average it? This is the real threat. In single views it may produce prettier shadows than our low-parametric model. The evidence against it is indirect. ObjectDrop itself reports shadows that conflict with true 3D light directions. Hu et al. and Dong et al. show that independent per-view denoising diverges and that reconstruction averages the conflicts into blur and floaters. We settle the question directly with two measurements. The first is shadow-layer MRC across views. The second is per-view shadow-mask error against real held-out-mover frames. If the 2D baseline matches us on both, the self-counterfactual idea adds nothing beyond consistency, and we would report that.
Why not just estimate the light once from a single image and ray-trace shadows of the inserted Gaussians? Outdoors, with one sun and hard shadows, this may work nearly as well. The static-cue ablation is exactly this method. Our claim is narrower. Indoors, with several lights, soft shadows, and bounced color, the movers' occlusion pattern over time constrains the light parameters far better than any single frame can. We will report results split by indoor and outdoor scenes. If the advantage appears only indoors, the claim will be scoped to indoor scenes.
Why not just do full inverse rendering of the dynamic scene? D3DR reports that R3DG takes 140–185 minutes and 0.955 GB per static insertion, against roughly 50 minutes and 0.076 GB for its own method. Material decomposition also remains brittle in the wild. We never estimate materials, only $K$ lights, an ambient term, and occlusion. The inverse-rendering baseline, run under an equal compute budget, will show whether the extra physical detail buys any accuracy in shadow regions.
Why not just collect paired dynamic insertion data and fine-tune end-to-end? Carve3D shows that supervised fine-tuning on small 3D datasets erodes realism, and that Instant3D-100K still trails an RL-tuned model. Paired dynamic multi-view captures with and without an object do not exist at scale. Our method needs no new collection, which is the whole point.
Proposed Experiments
We use the dynamic multi-view benchmarks in Split4D's suite (Multi-Human, Neural3DV, SelfCap) and the DyNeRF and Technicolor scenes from Instruct-4DGS. These contain people moving under real indoor and outdoor lighting.
The decisive protocol is held-out-mover re-insertion. In scenes with at least two movers, we fit the light model with one mover's Gaussians removed and its time window masked. We then re-insert that mover's reconstructed geometry and predict its shadows and shading. The ground truth is the real captured pixels. We score PSNR and LPIPS inside dilated shadow masks, plus a whole-frame score to verify that we do not damage the background. This protocol evaluates real light transport without synthetic data.
A complementary synthetic set uses rendered dynamic indoor scenes with known lights, built from BlenderKit and SceneNet assets in the spirit of D3DR's benchmark. It gives exact light-direction error, angular error in degrees, as a function of how many mover-frames are available. This produces a data-efficiency curve showing how much motion the method needs.
For new text-generated objects we report three further measures: shadow-layer MRC across views, time-extended MRC for temporal Janus, and a blind user preference study against the 2D insertion baseline. We also check that segmentation quality from Split4D's decomposition does not limit results by repeating the evaluation with ground-truth masks where available.
The experiment that would sink the idea is this: the static-cue ablation matches the full method on held-out-mover shadow error indoors. That outcome would mean the movers' counterfactuals carry no extra lighting information, and the core claim fails.
Acoustics is out of scope here. The same mover-as-occluder logic could constrain an audio-visual Gaussian field such as AV-GS, which is currently restricted to static scenes, but testing that requires paired dynamic binaural captures that RWAVS does not provide.
References
- ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion
- DreamBooth3D: Subject-Driven Text-to-3D Generation
- Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration
- VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis
- InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
- D3DR: Lighting-Aware Object Insertion in Gaussian Splatting
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separation
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
- Diffusion Model-Based Image Editing: A Survey
- Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
- ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis