A camera with its own light source, such as a phone flash, a robot's depth sensor or a headset's illuminator, has a fixed power budget. It can spread that power evenly over the whole view, or it can concentrate it on a small patch, where it overwhelms sunlight or sensor noise. Concentration works: packing light into narrow strips, sparse dots or steered spots recovers geometry in full sun and in near-darkness where uniform light fails. The catch is that concentration only helps if the light lands in the right place at the right moment. In a moving scene, "the right place" is where the scene will be a few milliseconds from now, not where it was in the last frame.
We propose closing that loop with a fast 3D scene model. We keep a continuously updated 4D Gaussian splatting model, which represents a scene as millions of small colored 3D blobs that can move over time and renders them in a few milliseconds. The model predicts the next frame, estimates where the prediction is least reliable, and steers the concentrated light there before the frame is captured. The frames that come back are then explained by a forward model that knows exactly where the light was sent. The lit patches therefore become measurements of surface color and depth, and are no longer mistaken for moving objects. Fast rendering is what makes this possible: the loop fits inside one frame only because a splatted 4D model can be rendered and queried within the frame budget.
Core Claim
Under a fixed optical energy per frame, a closed loop that predicts the next frame and steers concentrated light will reconstruct dynamic scenes in darkness and in direct sunlight with better novel-view quality and depth accuracy than two alternatives given the same energy. The first is open-loop concentrated patterns, such as fixed or rotating dot grids. The second is steering driven by the current passive frame. The advantage should grow as the energy budget shrinks and as objects move faster. Two ingredients carry the claim. One is a forward model in which the active light is a known, calibrated term. The other is using the model's own prediction, not the last observed frame, to decide where the light goes. Both must be tested separately, because either could turn out to be unnecessary.
Where This Sits
Three hardware lines establish that concentration beats spreading. Structured Light in Sunlight sweeps narrow laser strips and shows that the number of required measurements grows with the square root of ambient light, not linearly. It needs 32 images at 90,000 lux, and it warns that objects moving during the sweep tear the reconstruction. Seeing Far in the Dark with Patterned Flash packs a flash into a 1-in-16 dot grid that punches through read noise at about 3 m. It then fills in color and disparity frame by frame with a convolutional network, and it notes that sparse dots lose to uniform light at close range. Energy-Efficient Adaptive 3D Sensing "looks then leaps": it captures a passive frame, finds textureless regions and steers a phase-modulated laser only there, reporting a 99% reduction in optical power at 30 fps for about 100 points. These papers motivate the proposal. The last one also supplies its closest ancestor. The difference is that Tilmon et al. choose targets from the current passive image and keep no persistent scene model. That choice fails twice in our regime. In darkness the passive image is noise and cannot tell the system where to aim. Under motion, the target has moved by the time the light arrives.
A second line removes moving objects from static reconstructions: RobustNeRF with trimmed losses, NeRF On-the-go with learned uncertainty, and RobustSplat and 3DGS-HPC with delayed densification and patch-level classification. This line constrains the design. All of these methods assume constant lighting, so a spot that lights a static wall in one frame and not the next looks exactly like a transient object. They would either delete those photons or, as RobustSplat documents for plain 3DGS densification, spawn floating blobs to fit them. Explicitly modeling the light is therefore a requirement, not a refinement.
A third line supplies the method. GS-ProCams shows that projector-to-camera light transport can be written into differentiable Gaussian splatting, with per-blob albedo, a projector blur kernel, gamma and gain, at 297 FPS. It is explicitly limited to static, non-specular scenes. DeGauss splits a scene into static and deforming Gaussians and renders at 157 FPS. Split4D gives each Gaussian a velocity and learns them in a streaming, chronological order, which we need for online updates. We reuse the GS-ProCams light-transport term inside a DeGauss/Split4D-style dynamic model. The combination is new, but it is not the contribution in itself. The contribution is using that combined model as a controller. Active view-planning work such as AG-SLAM and ActiveSplat also uses a Gaussian model's uncertainty to choose the next measurement. However, it moves the camera through static scenes at roughly 8 fps. We keep the camera where it is and move the light, inside a 33 ms loop, in scenes that change.
Research Program
The forward model treats each captured pixel $p$ at time $t$ as
$$I_t(p) = A_t(p) + g\,\rho(x_p)\,\frac{\cos\theta_p}{d_p^2}\,(K * E_t)(x_p) + n_t(p),$$
where $A_t$ is the ambient image rendered by the Gaussian model, $\rho(x_p)$ is the albedo of the surface point seen at $p$, $\theta_p$ and $d_p$ are the incidence angle and distance from the illuminator, $E_t$ is the commanded illumination pattern, $K$ is the calibrated projector blur kernel, $g$ is gain, and $n_t$ is shot plus read noise. The pattern is subject to $\sum E_t \le P$ for a per-frame budget $P$. Because $E_t$ is known, a lit patch changes $\rho$ and depth rather than tripping a distractor detector. Genuine movers are still handled by the static/dynamic split. Because the illuminator is offset from the lens, the position of each lit spot also gives a triangulated depth constraint, as in Patterned Flash.
The controller runs each frame. It renders the model's prediction for the next capture at reduced resolution. It then scores each image patch by expected information gain, computed from three quantities: the predicted signal-to-noise ratio under ambient light alone, which is low in dark regions or ones swamped by sun; how many recent measurements support the Gaussians there; and whether those Gaussians belong to the moving component. Budget goes to the top-scoring patches, and the concentration factor is chosen from the measured ambient level, following the optimal block size rule of Gupta et al. We start with a greedy allocation and add a learned allocator only if the greedy one proves to be the bottleneck.
The online model only has to be good enough to aim the light. It is updated by a few streaming gradient steps per frame on recent frames. The final reconstruction is refined offline on the full sequence. This split is deliberate: the file reports that training dynamic Gaussian models still takes 15 to 60 minutes. The latency claim therefore applies to prediction and allocation, not to full training.
Evaluation starts in simulation and then moves to hardware. In simulation we re-render the D-NeRF synthetic dynamic scenes (D-NeRF) under a simulated steerable illuminator. Physical noise is modeled as in Patterned Flash, at ambient levels from near-dark to a simulated 50,000 to 90,000 lux, with ground-truth albedo and depth available. The hardware prototype pairs a machine-vision camera with a phase-modulator laser illuminator, following Tilmon et al. Honestly, only that hardware class truly concentrates power. A DLP projector only masks light, so any DLP experiments will be labeled as emulations with an energy accounting correction.
Baselines
The naive reference is uniform flash at the same per-frame energy, reconstructed with DeGauss. This shows what happens if nothing is concentrated. A second naive point uses uniform flash with low-light Gaussian reconstruction via LL-Gaussian, which brightens with diffusion priors at about 70 FPS on a V100. It stands in for "fix it in software."
The strongest existing pipeline is a cascade. Patterned Flash's per-frame network enhances each dotted frame, and the enhanced frames are then fed to DeGauss. This uses real concentrated photons and a real-time dynamic model, and it lacks only the coupling and the loop. DeGauss reports 157 FPS on an RTX 4090. The file lists no reconstruction-quality numbers for Patterned Flash or for any method on actively lit dynamic scenes. We found no reported result on Lacuna for any baseline in this exact setting, so all comparisons will be run by us. We will retrain the Patterned Flash network on our simulated noise model and give DeGauss the same iteration budget and GPU as our offline refinement.
Two more baselines address the adaptive part of the claim. One uses a fixed 1-in-16 dot grid that shifts each frame to cover the scene over time, reconstructed with our full forward model. This is the strongest open-loop competitor. The other is Tilmon-style look-then-leap steering from the current passive frame, also reconstructed with our forward model. The difference between these two baselines and our method isolates the value of prediction.
A third ablation removes the explicit light term and trains RobustSplat on the actively lit frames, so that the light is treated as a distractor. This isolates the forward model.
Beating the baselines means lower energy to reach a fixed held-out PSNR, or higher PSNR, SSIM and depth accuracy at equal energy. It must beat the shifting dot grid by a margin larger than the variance across scene seeds, and the result must hold at more than one motion speed. Clean well-lit D-NeRF training, which reports 31.29 dB on Mutant, is an upper reference, not a competitor.
Why not just…?
Why not just use a brighter uniform flash, or average more frames? The physics in the file already argues against both. Gupta et al. show that averaging needs linearly more frames as ambient light rises, against square-root growth for concentrated scanning. Tilmon et al. report that full-frame projection needs about 100 times the power, which eye-safety limits often forbid. Averaging also blurs moving objects. We still run uniform flash at equal energy as a reference, so the size of the gap is measured rather than assumed.
Why not just mask out the known lit pixels and run a distractor-robust splatting method on the rest? This is simple and removes the confusion. It also discards exactly the pixels that carry signal in darkness or sunlight. The RobustSplat ablation, together with a mask-out variant, will show how much quality depends on using those photons. If mask-out comes close in bright ambient light, that is an informative boundary: the forward model would then matter mainly when the scene is starved of light.
Why not just use a fixed, shifting dot pattern and skip the controller? This is the real threat. Open-loop coverage may be enough when the budget is generous or the motion is slow, and Tilmon et al.'s savings were measured against full-frame light, not against sparse dots. We will map energy-versus-quality curves across budget and object speed. If the shifting grid matches closed-loop steering across that range, the conclusion is that the explicit forward model is the contribution and adaptivity is not. The proposal would be rewritten around that result.
Why not steer from the current passive frame, as Tilmon et al. do? In darkness the passive frame carries no usable signal, and under motion it is one frame stale. The object-speed sweep tests the staleness argument directly, and the ambient sweep tests the darkness argument.
Proposed Experiments
The first check is a static sanity test on the GS-ProCams synthetic and real projector-camera data (Nepmap). Our light term, with dynamics switched off, must roughly match GS-ProCams' reported synthetic PSNR of 31.97 dB, or any later gain may come from implementation differences.
The main experiment is on the re-rendered D-NeRF scenes at three ambient levels and four object speeds. We plot held-out-view PSNR, SSIM and LPIPS against the ground-truth appearance under well-lit, ambient-free conditions, together with depth error and a floater count (Gaussians more than a fixed distance from the true surface), all against per-frame energy for every method in the Baselines section.
The decisive latency test injects artificial delay into the loop, from 0 to 3 frames. If quality does not fall with delay, rendering speed is not central, and the claim that fast rendering makes the loop possible is falsified. We also report measured prediction and allocation time per frame on an RTX 4090 and on an embedded board, against the 33 ms budget.
The hardware experiment repeats the equal-energy comparison of closed-loop steering, the shifting grid and uniform flash on walking people and hand-object scenes at about 2 to 3 m. It runs indoors in darkness and outdoors in sun, with a second calibrated camera providing held-out views.
References
- Structured Light in Sunlight
- Seeing Far in the Dark with Patterned Flash
- Energy-Efficient Adaptive 3D Sensing
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- GS-ProCams: Gaussian Splatting-Based Projector-Camera Systems
- DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstruction
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- AG-SLAM: Active Gaussian Splatting SLAM
- ActiveSplat: High-Fidelity Scene Reconstruction Through Active Gaussian Splatting
- D-NeRF: Neural Radiance Fields for Dynamic Scenes
- LL-Gaussian: Low-Light Scene Reconstruction and Enhancement via Gaussian Splatting for Novel View Synthesis