Shadows That Move With the Robot: Actor-Conditioned Shading for Gaussian Real-to-Sim

Robotics researchers increasingly test and train robot policies inside photorealistic "digital twins": 3D reconstructions of a real tabletop built with Gaussian splatting, a representation that draws a scene as millions of small colored, semi-transparent blobs that can be rendered very quickly. These twins split the world into a frozen background and a few movable parts, such as the robot arm and the objects it pushes. The split contains a quiet error. The background's appearance is baked in at scan time, so when the simulated arm or a pushed block moves, no shadow moves across the table beneath it, and no darkening appears where things touch. Real cameras see those shadows. We propose to make the frozen background respond to the moving parts. Each background blob learns how strongly it darkens when something sits between it and the scene's light. The light is approximated by a handful of directions inferred from the capture video. Shadows are computed on the fly from wherever the moving parts currently are. Because shading depends on actor geometry rather than on a timestamp, it extends to robot motions that never appeared in the capture.

The broader question is a live disagreement in the literature about whether the static part of a decomposed scene can be treated as free and frozen. ResWorld freezes an aligned static base and predicts only dynamic residuals. DDP-WM argues that freezing the background lets errors accumulate, and adds a cheap low-rank background correction. We test the same question in rendered pixels, where it bears directly on the gap between simulation and reality. We measure whether a frozen background measurably distorts policy evaluation, and whether a cheap correction driven by the moving actors removes that distortion at real-time cost.

Core Claim

Current Gaussian real-to-sim pipelines factor the scene into static and dynamic parts but give the static part no way to respond to the dynamic part. This missing interaction term is a specific, fixable source of visual mismatch between simulation and reality. We add a per-blob shadow-receptivity scalar $s_i$ and a small set of $K$ learned light directions $d_k$ with weights $w_k$. A static blob at position $x_i$ is then rendered with color

$$\tilde c_i(t) = c_i \Big(1 - s_i \sum_{k=1}^{K} w_k\, O_k\big(x_i;\, \mathcal{G}_{\text{dyn}}(t)\big)\Big),$$

where $c_i$ is its baked color and $O_k$ is the accumulated opacity of the dynamic blobs $\mathcal{G}_{\text{dyn}}(t)$ along direction $d_k$ from $x_i$. $O_k$ is read from a low-resolution shadow map that is re-splatted only from the few thousand dynamic blobs at each frame. The parameters are fit on the capture video, where real people, hands, or objects moved and cast real shadows. At simulation time the term is driven by the simulated robot and objects.

We claim three things. First, the term improves rendering fidelity in actor-shadowed regions. Second, it lets objects be removed cleanly without shadow scars. Third, and most important, it raises agreement between simulated and real policy success on tasks where the arm and the manipulated objects cast visible shadows. It should do all this while adding under roughly 20% to frame time.

Relation to Prior Work

Three works motivate the gap. Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions builds 3D Gaussian scenes from phone scans and couples deformable objects to a spring-mass simulator. It reports Pearson correlation r > 0.9 between simulated and real policy success, against r = 0.237 for IsaacLab on rope routing. Its background appearance is static. RoboSimGS renders the background with Gaussians and the interactive objects as physics-engine meshes. It reports zero-shot transfer on eight tasks, but with no actor-to-background lighting interaction. Split4D and NeRF On-the-go document the symptom from the reconstruction side. Removing objects leaves inconsistent shadows, and shadows and highlights get misclassified as transient clutter.

Two lines constrain the design. Distractor-robust methods absorb lighting change with global per-image color transforms or brightness masks: RobustSplat++ uses per-image affine color transforms, and DeGauss learns brightness control masks alongside a static/dynamic composition mask. These corrections are indexed by image or time. A sim trajectory that never occurred in the capture has no index, so they cannot extrapolate to it. Our term is indexed by actor geometry, and that is the central difference.

Two works supply the method. Fully Explicit Dynamic Gaussian Splatting promotes blobs from static to dynamic by their motion, and DeGauss separates static and dynamic blobs with a composition mask. We reuse such a decomposition as the starting point. The delayed-densification schedule of RobustSplat keeps early optimization from spawning floating blobs to explain moving dark regions. We keep it, so that shadows are explained by the shading term rather than by geometry.

Research Program

The work has three stages, and the first can falsify the whole project cheaply.

The first stage is a sensitivity study. Before building anything, we ask whether policies care. We take the released-style pipeline of the Real-to-Sim paper and evaluate the same imitation policies on a T-block pushing task and a rope routing task under two conditions. In one, sim frames have the baked static background. In the other, real shadows are pasted in by compositing: a shadow matte is extracted from paired real frames recorded along the same scripted trajectories. If success rates and sim-real correlation do not move, the visual side of the gap is not shadow-dominated, and we stop at a short negative-result report.

The second stage is the model. We start from a static/dynamic Gaussian decomposition trained with delayed densification. We add the receptivity scalars and $K$ (between 2 and 8) light lobes, fit by photometric loss on capture videos in which a person moves objects and an arm across the table. Two regularizers keep the shading term honest. A sparsity prior on $s_i$ prevents the term from explaining all appearance change. Darkening is also allowed only when $O_k > 0$, so it cannot absorb global exposure drift; a separate per-image affine term, as in RobustSplat++, handles that drift. The dynamic blobs that cast shadows include the robot, articulated with its kinematics, and the manipulated objects, which are driven by the physics engine.

The third stage is integration and latency. The $K$ shadow maps are splatted at low resolution from dynamic blobs only, then sampled by static blobs in the existing rasterizer. We profile end-to-end frame time on one consumer GPU. For reference, the Real-to-Sim paper reports 5–30 FPS on a single GPU, DeGauss reports 157 FPS on an RTX 4090, and Ex4DGS reports 62.1 FPS on an RTX 2080Ti.

Baselines

The naive reference is the frozen background: the Real-to-Sim pipeline as published, with physics, robot, cameras, and color calibration unchanged. The only difference from our system is the shading term, so this comparison isolates the contribution exactly. Its reported correlation is r > 0.9 across its suite (Real-to-Sim Robot Policy Evaluation). That leaves a ceiling problem, so we report correlation and also absolute per-policy success gaps and rank agreement across checkpoints. Beating it means a statistically significant reduction in mean absolute sim-real success gap, with bootstrapped intervals over episodes, on the shadow-heavy tasks.

The strongest appearance-model competitor is a global per-frame correction in the style of RobustSplat++. We give it the same capture videos, the same decomposition, and the same number of training iterations. At sim time it uses the nearest-in-time appearance code. On the NeRF On-the-go benchmark RobustSplat++ reports PSNR 21.08 against 19.65 for vanilla 3DGS. We run it on our own captures and on NeRF On-the-go so that numbers are comparable. Beating it requires higher masked PSNR inside held-out actor-shadow regions and a smaller sim-real success gap.

The strongest conventional-graphics competitor is a hand-placed shadow-mapping light. We pick a single directional light by hand from the scan and cast hard shadows from robot and object meshes, roughly what a RoboSimGS-style hybrid could add with standard rasterizer features. It receives the same physics and the same tuning time for the light placement.

The key ablations are $K=1$ against learned $K$, constant $s_i$ against learned per-blob receptivity, and the shading term trained on time-indexed rather than geometry-indexed input. The last ablation directly tests the extrapolation claim.

Why not just…?

Why not just randomize shadows during policy training? Pasting random dark blobs into training images could make policies indifferent to shadows. That would help transfer, but it does nothing for evaluation fidelity, which is what real-to-sim digital twins are mainly used for. It is still a real threat on the transfer side. We include a shadow-augmentation arm in the transfer experiment, and if it matches our method there, we will narrow the claim to policy evaluation and reconstruction quality.

Why not just use a per-image appearance embedding? RobustSplat++ and DeGauss show that such embeddings absorb global lighting change. They are keyed to capture frames, though, and a novel sim trajectory has no frame to look up. The time-indexed ablation shows this directly. We expect it to match our method on held-out capture frames from seen trajectories and to fall behind on new ones.

Why not just add a classical shadow-casting light to the mesh simulator? This is the most plausible competitor, and it may suffice under a single dominant lamp. Our bet is that real lab lighting is several soft sources, and that per-surface response varies with material, which Gaussian blobs capture but a hand light does not. The hand-placed-light baseline settles this. We will report scenes with one dominant source and scenes with diffuse multi-source lighting separately, and we expect gains to concentrate in the latter.

Proposed Experiments

Reconstruction fidelity uses three sources. We capture our own tabletop sequences, recorded both with and without actors, which gives ground-truth shadow-free backgrounds. We also use NeRF On-the-go, and the Neu3D sequences, where Ex4DGS reports 32.11 dB PSNR. Metrics are PSNR and LPIPS computed separately inside and outside actor-shadow masks. Also on N3V, a check against no reported result found on Lacuna for shadow-region PSNR on Neu3D: a gain in shadow regions with no loss elsewhere is the expected signature.

Object removal is scored against the actor-free captures. It tests whether turning off $O_k$ removes shadow scars of the kind Split4D reports.

The decisive experiment is policy evaluation on a physical arm. We reproduce T-block pushing and rope routing, plus a deliberately shadow-heavy variant under low-angle lighting, and evaluate ACT and Diffusion Policy checkpoints in real and in each sim variant. The prediction is specific. Our method closes the sim-real success gap on the shadow-heavy variant and is neutral under diffuse overhead light. Gains under diffuse light would point to a confound; no gain under low-angle light would falsify the claim.

Latency is reported as frame time with and without the term at matched resolution. We count the result as a failure if the overhead exceeds 20%, because a slower renderer undercuts the purpose of these twins.