A relightable 3D model is a reconstruction that can be shown under lighting it was never photographed in. Building one today usually requires a light stage, a dome of dozens to hundreds of lamps fired one at a time while cameras record every combination. The best such models are hybrids. Known optics, such as how shininess depends on the angle between light and viewer or how light diffuses beneath skin, are written as fixed equations. A small neural network then learns whatever those equations miss. This proposal asks a plain question. If you can afford only a tenth of the camera-and-lamp combinations, which ones should you photograph? Our answer is to pick, in batches, the combinations that most sharpen the model's estimates of its physical quantities: surface color, roughness, surface orientation, and how far light scatters inside the material. Each candidate is scored only on information that the neural correction network could not have explained away on its own.
The hybrid structure makes this cheap. Because the physical part of the renderer is a closed-form equation, we can compute in closed form how much a not-yet-taken photograph would pin down each physical parameter, without retraining the model for every candidate. When a batch is scored by the joint information it carries, rather than by adding up individual scores, the batch spreads itself out automatically. It does so in the space of what each photograph reveals, not in the space of lamp angles. Two lamps that both light up the same specular highlight count as nearly one photograph. A grazing lamp that exposes surface orientation in a region nothing has constrained counts for a lot.
Core Claim
Existing active-capture methods for neural 3D reconstruction choose only camera positions, with lighting fixed and baked into the result. This holds for ActiveNeRF, which picks views by the expected drop in per-pixel color variance. It holds for ActiveGS-CRP, which clusters a Gaussian-splat scene into regions and targets the regions where a point-cloud autoencoder's features vary most under dropout. It holds for PUN, which trains a feedforward network to predict reconstruction error over a sphere of viewpoints, and for ActiveSplat and DNRSelect, which plan robot or render-camera trajectories. None of them can choose lights, and none measures uncertainty over a physical decomposition.
On the rendering side, SSD-GS, Subsurface Scattering for Gaussian Splatting and BecomingLit get their quality from dense one-light-at-a-time capture, using 40 to 167 lamps. They also suffer from role leakage: the neural part absorbs shadows or highlights that should belong to the physical terms.
The claim is that a batch-selection rule built from the residual-marginalized information about the physical parameters does three things, compared with spreading lamps evenly over the sphere at the same budget:
- It reaches higher relighting quality on held-out lights.
- It reduces role leakage.
- Its gains concentrate on shiny and translucent materials, while its advantage disappears on matte ones.
The third prediction is what makes the claim falsifiable rather than a blanket "active beats uniform".
The Selection Rule
Some notation for a single query $q$, which is one (camera, lamp) pair:
- $\theta_p$ are the physical parameters of the current model: per-Gaussian albedo, roughness, normal, and diffusion-profile parameters.
- $\theta_r$ are the parameters of the neural correction network.
- $J_p(q)$ and $J_r(q)$ are the Jacobians of the rendered pixels with respect to $\theta_p$ and $\theta_r$.
The physical part is analytic, for example Cook-Torrance specular or the dipole profile in SSD-GS. So $J_p$ is exact and cheap to compute. For $J_r$ we take only the correction network's last layer.
The information that query $q$ carries about the physical parameters, after allowing the correction network to soak up whatever it can, is a Schur complement:
$$ S(q) = J_p^\top J_p - J_p^\top J_r \left(J_r^\top J_r + \lambda I\right)^{-1} J_r^\top J_p . $$
Here $\lambda$ is a ridge term standing in for the regularization on the correction network. A batch $B$ is chosen to maximize
$$ \log\det\Big(\Lambda + \sum_{q \in B} S(q)\Big). $$
$\Lambda$ is the information already accumulated from captured images plus a prior. Two properties follow:
- Diversity comes for free. Because log-det is submodular, greedy selection is near-optimal. It also discounts a second query whose information duplicates the first. The same mechanism is what makes determinantal point processes favor diverse subsets (Lin et al.).
- The Schur term targets role leakage. It is what separates this rule from simply applying Fisher-based view selection to lights. A photograph whose dark pixels could equally be explained by a shadow the network learns scores near zero, however high its raw color error.
To keep the computation tractable, we make $\Lambda$ block-diagonal over the scene regions from ActiveGS-CRP-style clustering and over the physical components. We also score only pixels that the candidate actually sees.
Selection follows the timing result of TCM. That paper found that, starting from good representations, pure diversity sampling is needed for only one to three initial rounds before uncertainty-driven sampling takes over. Our Jacobians are meaningless before geometry exists, so the first round is a small Fibonacci lattice: an evenly spread set of points on the sphere. The information rule takes over after that.
We also test one structural hypothesis from the file's Gap 4. In the first rounds, $\Lambda$ is restricted to the diffuse and shadow blocks; scattering and specular blocks are added later. This mirrors the diffuse-to-shadow-to-scattering-to-specular training order that SSD-GS found necessary to stop leakage.
Research Program
Backbones. We build the selector on two existing hybrid backbones, keeping their architectures unchanged:
- The Gaussian model from Subsurface Scattering for Gaussian Splatting. It stores physically based material attributes per Gaussian and predicts the scattered light with a small network. It reports about two hours of training, which makes repeated rounds affordable.
- SSD-GS. It splits appearance into diffuse, specular, shadow and dipole scattering.
Pool and budgets. The candidate pool is the full set of captured (camera, lamp) images in each dataset. Acquisition is simulated by revealing images, so every method draws from identical real data. Budgets are 5, 10 and 20 percent of the pool, acquired over four rounds.
Analysis. Beyond the main comparisons, we check whether the score is honest. For each round, we compute the rank correlation between each candidate's predicted information gain and the actual drop in held-out relighting error after retraining with that candidate. This is measured on a subsample where retraining per candidate is feasible. We also time the scoring step against retraining, since PUN showed that per-candidate retraining is the bottleneck that makes active capture impractical.
Baselines
Naive reference: even spacing. The naive reference is a Fibonacci lattice over lamps, crossed with evenly spread cameras, at the same budget. This is what a light-stage designer would actually do. It is a serious competitor, because in a two-dimensional lamp space even spacing already achieves near-perfect coverage. Random selection is reported alongside it. Beating even spacing means a statistically reliable relighting PSNR gain on held-out lamps, averaged over objects at the 10 percent budget, plus lower albedo and normal error on synthetic scenes.
Pure uncertainty. We port ActiveNeRF's rule to the joint pool: per-pixel color variance with top-k selection. This is the pure-uncertainty baseline that the redundancy-trap argument targets.
Strongest existing active method. This is ActiveGS-CRP, extended so that its region-targeted candidate sampling also proposes lamps. Its regions are clustered from geometry, texture and visibility, and it targets them with dropout feature variance. The strongest number it reports is 25.542 dB PSNR on NeRF-Synthetic with 10 views, nearly 2 dB above its active baselines. It also reports a 5.7 dB average gain over FisherRF on complex Objaverse objects. These are view-only results under fixed lighting. No reported result was found on Lacuna for any active method on a relighting benchmark, nor for any of the hybrid backbones trained on subsampled lights.
Fairness. Every baseline uses the same backbone and the same training iterations per round. Each gets the same hyperparameter search of eight configurations, tuned on one held-out object per dataset.
Diversity in the wrong space. We run a density-aware core-set in lamp-and-camera angle space, following DACS: density tiers with k-center selection, then filtering by uncertainty. DACS reports 92.1 percent against 88.5 percent for pure uncertainty on Repeated MNIST at 0.4 percent labels. It therefore stands for the best of diversity-aware active learning as practiced, but in the input space rather than in information space.
Ablations of the key ingredient.
- Remove the Schur complement and use $J_p^\top J_p$ alone. This isolates the role-leakage term.
- Replace batch log-det with top-k individual scores. This isolates batch diversity.
- Drop the staged diffuse-first block schedule.
Upper bounds. The full-pool results are the ceiling:
- Subsurface Scattering for Gaussian Splatting reports 37.35 dB on its synthetic scattering set.
- SSD-GS reports 31.12 dB on the real Pixiu object.
- BecomingLit reports 31.38 dB relighting PSNR on its faces.
Why not just…?
Why not just space the lamps evenly? This is the real threat, and the proposal is built so that it can lose to it. On a matte object, every lamp direction carries similar information, and an even lattice is close to optimal. The one support for active choice in the file is view-only: ActiveNeRF reports 28.51 against 28.04 dB over uniform-trained NeRF at 10 views. That margin is small and comes from a different setting. The argument for our rule is about materials. Specular lobes and subsurface scattering are informative only at narrow light-view geometries that an even lattice hits by chance. So we stratify results by material: diffuse, glossy, and translucent objects in the synthetic scattering set and in NRHints. If even spacing ties our rule on glossy and translucent objects too, the claim is false.
Why not just pick the most uncertain images? DACS shows pure uncertainty wasting budget on redundant samples. The file also documents a negative correlation (−0.71) between local density and prediction entropy, which means uncertainty seeks sparse regions. In rendering, the highest color errors sit on specular peaks and shadow edges. A top-k variance rule should therefore cluster lamps around a single highlight. The top-k ablation of our own score separates "information about the physics" from "batch diversity". The ActiveNeRF port shows what raw color variance does on its own.
Why not just apply a diversity-aware active learner such as DACS or a TypiClust-style method to the lamp and camera angles? On a low-dimensional sphere, angular diversity collapses into something close to even spacing. It also cannot tell that two very different lamp angles reveal the same thing, or that one direction resolves a shadow-versus-albedo ambiguity that others cannot. The angle-space core-set baseline is run exactly to test whether diversity in information space matters, rather than diversity as such.
Why not just train a feedforward predictor, as PUN does? PUN obtains a 400x speedup, but its predictor learns geometric cues on a viewing sphere. It has no access to instance-specific material behavior. Our scoring already avoids per-candidate retraining by using analytic Jacobians, so the speed argument shrinks. We report scoring time per round to make this concrete.
Why not just fix leakage with stronger priors or staged training on fewer lamps? Priors help, as PN-Relighting shows with parametric geometry on sparse lights. But they presume a template, and the file notes that templates fail on hair, glasses and teeth. Every baseline here, including even spacing, already uses SSD-GS's staged training. Any remaining gap is therefore attributable to what was captured, not to how the model was trained.
Decisive Checks
Datasets.
- The synthetic scattering set from Subsurface Scattering for Gaussian Splatting: five objects with 112 lights and ground-truth materials. This is the main testbed for decomposition error.
- The NRHints real captures used by SSD-GS.
- The multi-view OLAT face set from BecomingLit, as a transfer check on skin.
Metrics.
- Relighting PSNR, SSIM and LPIPS on held-out lamps and views.
- Albedo, normal-angle and roughness error where ground truth exists.
- A leakage measure: correlation between recovered albedo and the ground-truth shadow map.
What would confirm the idea. Three outcomes, together:
- At 10 percent of the pool, the selector beats even spacing and the ActiveGS-CRP extension on glossy and translucent objects, and ties on diffuse ones.
- Removing the Schur term specifically raises the leakage measure.
- Predicted information gain rank-correlates positively with realized error reduction.
What would refute it. Either of two outcomes:
- Even spacing matches the selector across material classes.
- The angle-space core-set matches it, which would mean diversity in information space adds nothing over diversity in angle space.
References
- ActiveNeRF: Learning Where to See with Uncertainty Estimation
- Active Learning of 3D Gaussian Splatting with Consistent Region Partition and Robust Pose Estimation
- PEERING INTO THE UNKNOWN: ACTIVE VIEW SELECTION WITH NEURAL UNCERTAINTY MAPS FOR 3D RECONSTRUCTION
- ActiveSplat: High-Fidelity Scene Reconstruction Through Active Gaussian Splatting
- DNRSelect: Active Best View Selection for Deferred Neural Rendering
- SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting
- Subsurface Scattering for Gaussian Splatting
- BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading
- Long-Tailed Recognition via Information-Preservable Two-Stage Learning
- Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training
- In Defense of Core-set: A Density-aware Core-set Selection for Active Learning
- Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset