Stochastic Emitter Selection for Inverse Rendering from LDR Photos

When you photograph a room with an ordinary camera, lamps and windows blow out to pure white. A glossy tabletop reflecting a lamp, or a white wall in sunlight, can blow out in the same way. Recovering the room's true lighting and materials from such photos is called inverse rendering. It requires deciding which bright regions are actual light sources, which is a yes-or-no choice for each region. It also requires estimating continuous quantities: how bright each source is, what the surfaces are made of, and what curve the camera used to compress brightness into 8 bits. Today's best method for indoor scenes makes the yes-or-no choice once, with a fixed saturation threshold, and then optimizes everything else by gradient descent. If that first decision is wrong, the rest of the optimization inherits the error and cannot undo it.

We propose making the discrete choices part of the same optimization as the continuous ones. We keep a probability for every candidate light source and for every family of camera tone curve. At each step we sample a few discrete configurations, render them with a differentiable path tracer, and update two things at once. Surface and lighting parameters move by ordinary gradients. The discrete probabilities move by a natural-gradient rule borrowed from neural architecture search. A floor on each probability keeps any candidate from being ruled out too early. The result is one optimization loop that can reverse an emitter decision whenever later evidence contradicts it. It costs roughly the same as one inverse-rendering run, not one run per candidate configuration.

Core Claim

Emitter identity and tone-curve family should be treated as random discrete variables whose distribution is learned jointly with continuous scene parameters. On scenes where glossy highlights and bright white surfaces saturate like light sources, this should improve emitter identification, recovered emitter radiance and albedo, compared with a threshold-then-optimize pipeline. It should also beat continuous relaxations of the same choices, which can spread a little emission across many surfaces and trade it off against the tone curve and albedo. The claim is falsifiable: if a well-tuned continuous relaxation matches our method, the discrete machinery is not worth its complexity. The experiments are built to detect that outcome.

What Exists and What Is Missing

The setting comes from IRIS. IRIS models 8-bit image formation with a learnable camera response curve constrained to a PCA subspace built from 201 real cameras. It flags emitters as pixels that stay saturated across views, then recovers their un-clipped radiance by path tracing against unsaturated secondary reflections. IRIS motivates this proposal because its detection step is a single hard threshold that later stages cannot revise. It also constrains the design: IRIS assumes a fixed mesh, and so will we. Geometry recovery is out of scope.

DiPIR optimizes Spherical Gaussian lights and a spline tone curve together in Mitsuba 3, a differentiable renderer, guided by a diffusion prior. It reports that lighting and tone curve can compensate for each other and produce wrong solutions. The number of lights in DiPIR is fixed in advance. HDR-Plenoxels self-calibrates exposure, white balance and a piecewise-linear response curve inside a voxel grid, using only continuous parameters. Pano-NeRF recovers clipped window radiance from light those windows reflect off diffuse surfaces. Across all four, every discrete decision is either fixed beforehand or relaxed into a continuous one.

The optimization machinery comes from three papers. ASNG-NAS trains network weights by gradient while updating a categorical distribution over architectures with an adaptive natural gradient. It samples discrete sub-networks rather than mixing all candidates, and searches CIFAR-10 in 0.11 GPU days where DARTS-style mixtures take 1.5 to 4. CatCMA and CatCMAwM contribute the margin: a lower bound on categorical probabilities that prevents a choice from being fixed permanently before the continuous parameters have settled. We borrow the margin, but not CatCMA's evolution strategy for the continuous parameters. That strategy needs hundreds to thousands of evaluations, and here gradients from the renderer are available for free.

The only existing paper that pairs tone mapping with black-box optimization is Tone Mapping via Meta-Guided Bayesian Optimization. It works on 2D HDR images with a known image and no scene model, so it does not bear on emitter recovery. Our proposal differs from all of the above in one way: the uncertain discrete structure of the scene and camera lives in a sampled distribution inside the gradient loop.

Research Program

The input is a set of calibrated multi-view 8-bit photos and a fixed mesh, as in IRIS. We first generate a deliberately generous set of $K$ candidate emitter clusters. These are groups of mesh faces whose projections are saturated or very bright in any view, found with a loose threshold, so that the candidates are a superset of the true emitters. Each candidate gets a binary variable. A single categorical variable chooses the tone-curve family from three options: the IRIS PCA response basis, a gamma-plus-offset curve, and a sigmoid "filmic" spline. The categorical choice matters because a PCA basis fails when a camera's real curve lies outside the training cameras. Continuous parameters $\theta$ cover per-cluster emission, per-face albedo and roughness, per-image exposure, and the coefficients of every tone-curve family.

Write $z$ for the discrete configuration and $p_\phi(z)$ for a product of independent Bernoulli and categorical distributions with parameters $\phi$. We minimize

$$J(\theta,\phi)=\mathbb{E}_{z\sim p_\phi}\big[\mathcal{L}(\theta,z)\big],$$

where $\mathcal{L}$ is the photometric loss between observed 8-bit pixels and the tone-mapped, clipped rendering of the scene under configuration $z$. Clipping is handled with a leaky saturation mask in the style of HDR-Plenoxels.

Each iteration proceeds as follows. We draw two configurations $z_1, z_2$ and render a minibatch of rays for each at low sample counts in Mitsuba 3. We take a gradient step on $\theta$ averaged over both samples; a parameter that is inactive in a given sample receives no gradient from it. We then update $\phi$ with the ASNG ranking-based natural gradient, and clip every probability to the interval $[\alpha, 1-\alpha]$ following the CatCMA margin. At convergence the configuration with the highest probability is taken as the answer. Continuous parameters are then briefly fine-tuned with that configuration fixed.

Three design questions will be studied explicitly. The first is how the margin $\alpha$ should scale with $K$, given the known degradation of fixed margins in low dimensions reported by Watanabe. The second is whether continuous parameters belonging to inactive candidates should be frozen or should decay. The third is whether two samples per step suffice.

Baselines

The naive reference sets every candidate cluster to be an emitter and uses a fixed gamma-2.2 curve, then runs the same gradient optimization. This shows how much any selection mechanism is worth.

The strongest existing method is IRIS. We run it on the same meshes, images and path-tracing budget, and allow its saturation threshold to be tuned on a held-out scene with the same number of tuning trials that we give $\alpha$. The strongest number we can cite is its albedo PSNR on synthetic indoor scenes, which rises "from approximately 15 dB to over 22 dB" against earlier LDR methods (IRIS). We found no reported result on Lacuna for emitter detection accuracy or emitter radiance error on this benchmark, so we will measure IRIS on those metrics ourselves. Beating IRIS means clearly higher emitter F1 and lower log-radiance error on the stress scenes described below, without losing more than a fraction of a dB of albedo PSNR on its original scenes.

The most dangerous competitor is continuous relaxation, run in two variants. In the soft-emission variant, every candidate gets a nonnegative emission parameter with an L1 sparsity penalty, and the tone family is replaced by a learned softmax mixture of the three curves. In the Gumbel-softmax variant, the discrete variables are relaxed with an annealed temperature. Both use the same renderer, iterations and tuning budget as our method, with the sparsity weight and temperature schedule tuned as fairly as $\alpha$.

A budget-matched outer loop tests whether "one loop" matters. CatCMAwM proposes discrete configurations, and each configuration gets a truncated inner gradient run. Total rendered rays are matched to our method.

Two ablations isolate the key ingredients. One removes the margin. The other removes the tone-family categorical and keeps only the PCA curve.

Why not just…?

Why not just relax the choices and use gradient descent, for example L1-sparse soft emission? This is the real threat, and we treat it as one. The failure we expect is the one DiPIR reports: when there are continuous knobs for both lighting and tone curve, faint emission spread across glossy or white surfaces can be offset by bending the tone curve and lowering albedo. An L1 penalty trades that error against sparsity rather than removing it. The mixed-variable literature adds that relaxed discrete variables create flat regions where gradients vanish (Watanabe; CatCMAwM). ASNG-NAS also found that sampling beats mixing on cost. None of this is evidence from inverse rendering, however. The soft-emission and Gumbel baselines decide the question directly. If either matches our method's emitter F1 and radiance error on the stress scenes, we will report that the relaxation suffices.

Why not just run IRIS with a tuned threshold or several restarts? A single threshold cannot separate a clipped glossy highlight from a clipped lamp when both saturate in every view. Restarting does not help, because the error is deterministic and not a matter of bad initialization. We will give IRIS a threshold sweep over the same trial budget. The stress scenes are built so that no single threshold is correct for every cluster, which is exactly the regime where a threshold-then-optimize method must fail.

Why not just wrap the pipeline in off-the-shelf mixed-variable Bayesian optimization or CatCMA? An outer loop pays for a full or truncated inverse-rendering run on every candidate. CatCMA-style methods need hundreds to thousands of evaluations (CatCMAwM). Bayesian optimization tools such as TPE and SMAC3 are about 100 times slower per iteration on SphereIntCOM, and their cost grows badly as $K$ increases (CatCMA). The budget-matched CatCMAwM baseline measures this cost directly, and our prediction is that truncated inner runs give it noisy estimates.

Why not just train on RAW sensor data? RAW training (NeRF in the Dark; LE3D) removes the tone curve. It does not remove clipping at lamps, and casual captures and archives are 8-bit JPEGs. The problem we target exists precisely when RAW data is unavailable.

Proposed Experiments

The main testbed is the IRIS synthetic indoor suite, extended with controlled stress variants. We add glossy surfaces that reflect lamps, near-white diffuse walls under strong light, and dim or partly occluded emitters that never saturate. Each variant is scripted so that its expected effect on threshold detection (false positive or false negative) is known in advance.

Tone-family selection is tested on the five HDR-Plenoxels synthetic scenes (Book, Classroom, Kitchen, Palace, Room), re-rendered through curves that lie outside the 201-camera PCA basis. Real-world transfer is checked on the Replica and iPhone LiDAR panoramas used by Pano-NeRF, scored with HDR-VDP3 and relighting PSNR. Pano-NeRF reports 9.67 HDR-VDP3 there, although its setting differs from ours.

Metrics are emitter F1 over candidate clusters, log error of emitter radiance, albedo and roughness PSNR, error of the recovered response curve, relighting PSNR, and wall-clock time. Every configuration is run over five seeds.

The decisive check is a comparison on the stress scenes. The method succeeds if it improves emitter F1 and radiance error over both IRIS and the best-tuned relaxation, matches IRIS on the original scenes, and runs in under 1.5 times IRIS's wall-clock time. If the relaxation matches us, we conclude that the discrete machinery is unnecessary. If removing the margin causes early lock-in on the dim-emitter scenes, the margin is shown to be the ingredient that matters.