Lighting Bases for Gaussian-Splat Simulators: Continuous Appearance Interpolation at Baked-Scene Speed

Robots trained in simulation often fail in the real world because the simulator's images do not look like real camera images. A recent fix is to rebuild the robot's actual workspace with 3D Gaussian splatting, a fast renderer that represents a scene as millions of small colored, semi-transparent blobs. These digital twins look real, but each one is frozen under a single lighting condition. Changing the lighting today means either running a neural network for every frame, which is slow, or switching among a few precomputed appearances, which gives discrete jumps. We propose giving every blob a small set of learned "lighting basis" colors, so that the scene's appearance is a linear function of a short latent vector. Physics supports this choice: rendered radiance is linear in the intensities of the light sources. Moving along the latent vector then corresponds to physically mixing lights. It costs a few multiply-adds per blob, so continuous lighting changes render at nearly the speed of a static scene.

Two practical gains follow from this one structural choice. During policy training, the simulator can draw a new, physically plausible lighting condition for every frame at no rendering penalty. During deployment, the robot can estimate the current real lighting from a single camera frame by solving a small least-squares problem rather than running iterative network optimization. Latent interpolation is the core mechanism, low rendering latency is the constraint that shapes its form, and transfer from simulation to reality is the setting where it is tested.

Core Claim

Appearance latents in existing splatting methods are either baked to one fixed setting or evaluated per frame by a neural network. WildGaussians (WildGaussians) learns per-image appearance embeddings and reaches 108–117 FPS only by folding one chosen embedding's affine color change into the blobs, which removes the network. DAVIGS (DAVIGS) keeps a network at runtime and uses a cell-based query to reach 151 FPS. Both treat the embedding as a way to absorb nuisance variation in tourist photo collections. Neither asks what lies between embeddings or uses them to generate training data.

We claim that restricting appearance to be linear in a nonnegative latent loses little fidelity on real lighting variation. With this restriction, every point between captured conditions is exactly renderable and meaningful. The restriction also makes appearance cheap to sample during training and cheap to invert at deployment. Concretely, blob $i$ seen from direction $d$ under latent $z \in \mathbb{R}^K_{\ge 0}$ has color

$$c_i(z, d) = c_i^0(d) + \sum_{k=1}^{K} z_k\, b_{i,k},$$

where $c_i^0$ is the usual view-dependent base color, $b_{i,k}$ is a learned RGB basis color, and $K$ (around 4–12) is the number of lighting factors. A global per-image camera response $g$ is applied after rendering, handling exposure gain, white balance and a gamma curve. It is a per-pixel operation, so it is also cheap.

Because alpha-blending is linear in color, the rendered image for a fixed camera pose is an affine function of $z$ before $g$ is applied. That property is what makes closed-form inversion at deployment possible.

The connection to the interpolation literature is direct. On Linear Interpolation in the Latent Space of Deep Generative Models analyzes why straight lines between latent codes can pass through regions the decoder never learned. Our decoder is linear by construction, so the convex hull of captured lighting codes contains no such low-density regions. The remaining question is whether that hull covers the real lighting the robot will meet, and that question is empirical.

Research Program

Capture and fitting. We capture each robot workspace under several real lighting conditions, for example switching individual lamps, opening blinds, and recording at different times of day. Each capture image receives its own $z$ and camera response.

The scene is fit with a standard splatting loss plus three regularizers. The first enforces nonnegativity on $z$. The second is a sparsity penalty on $z$, which encourages bases to align with individual light sources. The third is an identity-style penalty, in the spirit of DAVIGS's $L_{ID}$, that keeps $c_i^0$ close to a neutral appearance.

We keep transient-object masking from WildGaussians, which detects moving objects such as passers-by through feature discrepancy and excludes them from the loss. Exposure is assigned to $g$ and not to $z$, because exposure changes otherwise get absorbed as fake light sources. Separating them is one of the disentanglement questions we measure.

Interactive objects. Following RoboSimGS (RoboSimGS) and RE³SIM (RE³SIM), objects the robot touches remain meshes inside a physics engine. Unlike those systems, we render the objects with blobs attached to the mesh surface, as FlashAvatar (FlashAvatar) does for faces. Their basis colors are fit from the same multi-lighting captures, so lighting on objects and background changes together.

This is a known approximation. A basis fit at an object's rest pose does not move shadows correctly when the object moves. We quantify the error on held-out frames where objects are displaced.

Sampling lighting during training. We fit a density over the training codes $z$, starting with a simple mixture model. In the spirit of Neural Posterior Domain Randomization, which replaces hand-chosen randomization ranges with a distribution inferred from real data, we sample from this density restricted to the convex hull, plus a controlled margin outside it. The camera-response parameters are sampled separately.

Inverting lighting at deployment. For a known or tracked camera pose $p$, let $A(p)$ hold the $K$ basis renders and $a_0(p)$ the base render. The current lighting estimate is

$$z^\star = \arg\min_{z \ge 0} \lVert g^{-1}(I_{\text{real}}) - a_0(p) - A(p)\,z \rVert^2,$$

a nonnegative least-squares problem in $K$ unknowns. This mirrors the closed-form initial alignment in Found-adapt, but it is applied to the visual domain rather than to proprioceptive dynamics. It plugs into photometric pose tracking of the kind used in Vision-Only Robot Navigation in a Neural Radiance World. That system's map is static and has no mechanism to absorb lighting change.

Analysis. We will measure the following.

  • Fidelity versus $K$: how reconstruction quality changes as the number of lighting factors grows.
  • Coverage: how much of the real lighting variation lies inside the convex hull of captured codes.
  • Disentanglement: whether bases align with physical lights, and whether exposure stays in $g$ rather than leaking into $z$.
  • Linearity limits: where the linear assumption breaks, such as strong specular highlights, or effects like haze that are not mixtures of lights. We expect it to break there and will report those cases rather than hide them.

Baselines

Do-nothing reference. A single static splat of the workspace with one fixed appearance, the configuration used by RE³SIM's rendering of backgrounds. RE³SIM reports above 58% average zero-shot real success on its tabletop tasks: 75% bottle pick-and-drop, 25% block stacking and 75% vegetable placement, with a 0.924 correlation between simulated and real performance. It is run on our scenes with identical policy architecture (Action Chunking Transformer), demonstration count and training steps. Our method must beat it under real lighting that differs from the capture used to build the static splat. Matching it under the capture lighting is the minimum requirement.

Heuristic randomization. Uniform color jitter, brightness and contrast shifts, and randomized point lights on the mesh objects, at a tuning budget matched to ours (the same number of hyperparameter trials). PNDR (Photo-realistic Neural Domain Randomization) is the evidence that this matters: its physically grounded lighting randomization reached 40.44 on HomeBrewedDB Scene 10, against 22.24 for a model trained on real data with mismatched lighting. Dynamic lighting alone contributed a 19% accuracy gain. We therefore expect a heuristic baseline to be competitive and will not treat it as a straw man.

Strongest existing appearance-conditioned splatting. WildGaussians and DAVIGS, retrained on our multi-lighting captures. On fidelity, DAVIGS reports 28.31 PSNR on GLAV against 27.23 for GS-W and 22.80 for vanilla splatting, at 151 FPS versus GS-W's 24 FPS. For WildGaussians, no reported PhotoTourism PSNR was found on Lacuna, so we will reproduce it. Our method beats this baseline if it stays within about 0.5 dB of DAVIGS on GLAV and held-out lighting. It must also render continuously varying appearance at least twice as fast as DAVIGS's network-at-runtime mode on the same GPU, with VRAM growth under 50% over vanilla splatting.

Ablation of the key ingredient: discrete baked anchors. WildGaussians-style baking applied separately to each training-image appearance. Each episode samples one baked appearance; there is no interpolation. This isolates whether continuity matters or whether a library of captured appearances is enough. A second ablation removes nonnegativity and allows unconstrained $z$, to test whether the physical constraint helps transfer or only constrains fitting.

Why not just…?

Why not just bake a few dozen discrete appearances and sample among them? This is the most serious alternative. It matches our rendering speed, and the cost of $N$ baked color sets is comparable to $K$ basis colors. Our method contains it as a special case, so the contest is about coverage. If real deployment lighting mostly resembles one captured condition, discrete anchors will do as well. If it is a mixture, for example a desk lamp on while daylight fades, interpolated samples should help.

The discrete-anchor ablation, evaluated on real test conditions deliberately chosen as mixtures and as unseen single lights, settles this. If it matches us on policy success, we will report that continuity adds nothing for policy transfer. The contribution would then reduce to the fast inversion at deployment, which discrete anchors cannot provide in closed form.

Why not just run DAVIGS's network every frame with interpolated embeddings? At 151 FPS this may be fast enough for one camera. Demonstration generation at RoboSimGS scale, over 10,000 demonstrations per day on one consumer GPU, multiplies cameras, environments and frames, so per-frame overhead compounds. Interpolating an unconstrained network's embeddings also has no guarantee of producing physically plausible images, which is exactly the concern raised in On Linear Interpolation in the Latent Space of Deep Generative Models.

We test both claims directly. We measure throughput at multi-camera batch sizes. We also score interpolated DAVIGS renders against real mixed-lighting photographs, compared with our renders at the matching $z^\star$.

Why not just apply 2D image augmentation or image-to-image translation to the static splat's renders? 2D augmentation cannot move a shadow or brighten one side of an object. Global translation also ignores small, task-relevant regions: Leveraging Local Domains for Image-to-Image Translation showed that CycleGAN misses localized features. PNDR found that Pix2Pix and CycleGAN degraded 6D pose transfer relative to rendering-based randomization.

The heuristic-randomization baseline includes strong 2D photometric augmentation, so the policy experiment measures the shortfall rather than assuming it.

Proposed Experiments

Experiment 1: rendering fidelity, coverage and speed. We evaluate on GLAV and PhotoTourism, plus NeRF On-the-go for robustness to moving objects. We also build a new multi-lighting tabletop capture in which some lighting conditions are held out: interpolated mixtures of seen lights, and wholly unseen lights. Only $z$ and $g$ are fit on held-out images, with the scene frozen, and we report PSNR as a function of $K$ against WildGaussians, DAVIGS and vanilla splatting. We also report FPS and VRAM, both static and while $z$ changes every frame.

The decisive check is held-out mixture lighting. If our renders at $z^\star$ do not beat the best single training appearance by a clear PSNR margin, the linear-lighting premise fails for these scenes.

Experiment 2: zero-shot policy transfer. We use RE³SIM-style tabletop tasks (bottle pick-and-drop, block stacking, vegetable placement) with scripted expert demonstrations. Five training regimes are compared: fixed appearance, heuristic randomization, discrete anchors, continuous latent sampling, and continuous sampling with an extrapolation margin. Real trials, at least 20 per task per lighting condition, cover capture-like lighting, mixtures and unseen lights.

Metrics are success rate, the drop in success relative to capture lighting, and the correlation between simulated and real success, compared against RE³SIM's reported 0.924. The decisive check is the comparison against discrete anchors under mixture and unseen lighting. That gap is the evidence that continuous interpolation itself, rather than multi-lighting capture alone, helps transfer.

Experiment 3: pose tracking under changing light. A camera moves through the workspace while real lights are switched and dimmed. We compare three trackers. The first uses a static map. The second optimizes a WildGaussians-style embedding by gradient steps each frame. The third interleaves our closed-form $z^\star$ solve with photometric pose updates.

Metrics are pose error, divergence rate, and per-frame cost of estimating appearance, against a target of 25 Hz motivated by RNR-Nav's (RNR-Nav) 25.4 Hz real-time localization. The decisive check is whether the closed-form tracker avoids divergence after sudden lighting changes at lower per-frame cost than the gradient-based tracker. If it does not, the deployment half of the claim is withdrawn and the contribution is limited to faster training-time randomization.