Synthesizing High-Ambient Flash/No-Flash Training Pairs from Real Raw Captures

Many cameras separate a controlled light source (a flash, an infrared projector) from the surrounding room or sunlight by taking two photos, one with the source on and one with it off, and subtracting them. Light adds linearly in the camera's raw sensor values, so the average ambient contribution cancels. Photon noise does not cancel. Its variance adds across the two frames, so in bright sunlight the flash-only difference image is buried in noise and clipped pixels. Every method downstream of that subtraction then degrades, including reflection removal, color constancy, and structured-light depth. Training a network to recover the controlled-light layer in this regime needs many paired examples at high ambient-to-signal ratios. Today those examples come from expensive sources: weeks of ray tracing, multi-GPU diffusion sampling, or custom strobed hardware rigs.

This note proposes a data generator that costs almost nothing to run. It starts from existing real, well-exposed raw flash/no-flash pairs. It then uses the same additivity that makes subtraction work, together with an exact rule for re-synthesizing photon noise, to turn each pair into thousands of physically correct high-ambient, saturated, and slightly misaligned training pairs. The claim is that a small feed-forward network trained only on these pairs will recover flash-only images under strong ambient light better than models trained on rendered or generically noised data. It should also cost orders of magnitude less to generate data for, and should transfer to real outdoor captures without any real high-ambient training data.

What is new relative to existing work

The closest work is LightLab, which already uses "light arithmetic" ($I_{\text{on}} - I_{\text{off}}$ in linear color) to expand 16,000 Blender renders into about 600,000 relighting images. So using light additivity to cut synthetic-data cost is not new, and this proposal does not claim it. LightLab's arithmetic works on clean renders plus 600 real pairs. It recombines light layers to vary appearance and treats sensor noise as something the real pairs will teach. It is confined to indoor scenes and targets a diffusion model with roughly 5 seconds of inference per image. It does not model the noise and clipping that make high-ambient subtraction fail, and those are exactly the target here.

The structured-light simulator in Close the Sim2real Gap supplies the constraint. It shows that simulating the full sensing chain beats heuristic domain randomization: detection IoU of 0.648 against 0.584. Its cost is ray tracing, which it cuts from weeks to days with an AI denoiser. That denoiser removes the very shot noise that governs differential imaging in bright ambient light.

Neural Fields for Structured Lighting motivates the problem. It separates ambient from projector light with strobed pairs but reports degradation under strong sunlight, and it needs hours of optimization per scene.

The novelty therefore lies in two things. The first is the generator's noise and saturation model, which is exact for the pair and works from real captures instead of renders. The second is the high-ambient regime it targets.

Core mechanism

Work in raw units of photo-electrons, after subtracting the black level and dividing by the gain. A captured pair gives an ambient layer $a$ (the no-flash frame) and a flash layer $c = I_{\text{on}} - I_{\text{off}}$. Under the standard Poisson-plus-Gaussian sensor model, each observed pixel's noise variance equals its mean electron count plus a read-noise term $r^2$.

A naive way to simulate brighter ambient light is to scale up the captured ambient frame. That fails, because scaling a noisy frame by $k>1$ scales its variance by $k^2$, whereas a real frame $k$ times brighter has variance only about $k$ times larger.

The generator instead holds ambient light at its captured level and moves the ambient-to-signal ratio by scaling the flash layer down by $\alpha<1$, which is what inverse-square falloff or a weaker emitter does physically. Scaling down leaves too little noise, and that deficit can be topped up exactly. The synthetic on-frame is

$$\tilde I_{\text{on}} = \operatorname{clip}_{W}\big(I_{\text{off}} + \alpha c + \epsilon\big), \qquad \operatorname{Var}(\epsilon) = (\alpha - \alpha^2)\,\hat c + (\alpha - \alpha^2)\,r^2 ,$$

where $\hat c$ is a denoised estimate of the flash layer and $\operatorname{clip}_W$ saturates at a simulated full-well capacity $W$ chosen relative to the ambient level. The added variance brings the scaled flash layer's noise from $\alpha^2$ up to the $\alpha$ a genuinely weaker flash would produce. The pair's two noise terms are handled separately: the off-frame keeps its real noise, and the added noise enters only the on-frame. The difference image therefore carries the correct coupled variance, roughly $2a + \alpha c$, rather than two independent synthetic draws.

Two further cheap augmentations complete the generator. The first applies small homographies and depth-dependent parallax between the two frames, using monocular depth, to mimic handheld capture. This addresses the non-rigid and parallax failures of optical-flow alignment reported by Robust Reflection Removal with Flash-only Cues in the Wild. The second applies spatially varying ambient gain maps, which mimic sun patches but may only increase ambient where the base frame is clean. Each synthetic pair takes milliseconds of CPU time.

Research Program

The source data is the crowdsourced flash/ambient dataset: about 2,700 12-bit linear raw pairs from diverse scenes. Its value here is that each pair is a real, well-exposed decomposition. Sweeping $\alpha$ over two to three orders of magnitude, together with $W$, the warps, and the gain maps, yields effectively unlimited training pairs.

The learning target is the clean flash-only layer $\hat c$ at the original exposure. The model is a compact U-Net that takes both raw frames and predicts the clean flash-only layer. The model is deliberately small, so that any gains can be attributed to the data rather than to capacity.

The first analysis measures how the generator's statistics match reality. We will capture a small real calibration set on a tripod: 20 to 30 indoor and outdoor scenes, some through windows, with the flash fired at a fixed distance under ambient levels ranging from dim rooms to direct sun. For each scene we average 64 or more on/off pairs to get a low-noise reference flash layer. This set is used only for testing.

The second analysis is downstream. The recovered flash-only image replaces the raw difference as input to the reflection-removal network of Robust Reflection Removal With Reflection-Free Flash-Only Cues.

The third analysis accounts for cost. We report downstream accuracy against GPU-hours spent generating data, and place our generator alongside the reported costs of the alternatives: weeks of ray tracing in Close the Sim2real Gap and 62 hours on 8 GPUs for diffusion-generated training data in Zero-shot Generation of Training Data with DDPM.

Baselines

Naive reference. The do-nothing reference is raw subtraction, $I_{\text{on}} - I_{\text{off}}$, followed by optical-flow alignment for handheld pairs. This is exactly what Lei et al. feed their networks. Our model beats it if it improves PSNR on the real calibration set at every ambient bin. Improvement matters most in the top two bins, where subtraction is noise-dominated.

Off-the-shelf raw denoiser. A generic raw denoiser applied to the difference image is trained with the same U-Net on the same 2,700 base pairs. The only change is the noise: independent Poisson-Gaussian noise added to each frame, with brightness scaled up in the usual way. This is the strongest competitor on the core question, because it is the standard synthetic-noise recipe and costs the same to generate. We match its architecture, training steps, and hyperparameter search budget to ours. Beating it by a consistent margin in the high-ambient bins would isolate the value of the exact scale-down-and-top-up construction.

Rendered pairs with light arithmetic. A LightLab-style rendered corpus serves as the second competitor. We build flash/no-flash pairs from Blender scenes with light arithmetic and Gaussian noise, and match the generation budget in GPU-hours. We would then report accuracy at equal generation cost and at equal pair count.

Ablations. Separate ablations remove one ingredient at a time: the saturation model, the coupled noise (replaced by independent per-frame noise), and the warps.

Downstream reflection removal. The strongest reported numbers are 29.76 dB PSNR on FOR-Aligned for flash-only cues in the wild, and 32.06 dB for Flash-Split, a dual-branch latent-diffusion method. Both were measured in the benchmark's own ambient conditions. We found no reported result on Lacuna for any method on FOR under elevated ambient-to-flash ratios, or for flash-only layer recovery under high ambient light. We will therefore degrade FOR by the physically correct procedure above, construct a new high-ambient split, and rerun both released methods on it with and without our front end.

Success means two things. First, the gap between these methods and our front end should widen as ambient light increases. Second, at the original ambient level we should stay within about 0.5 dB of their published numbers.

Why not just…?

Why not just capture real high-ambient pairs? A real reference needs a tripod, a static scene, and many averaged exposures. It cannot scale to thousands of diverse scenes, which is the hardware bottleneck noted across the active-sensing literature. We do capture such data, but only as the test set. If a model fine-tuned on half of it matched ours, the conclusion would be that the generator's value lies only in avoiding capture effort. That is a weaker result, and we would report it as such.

Why not just use standard synthetic raw noise? This is the most serious threat, because it is nearly free and widely used. The coupled, variance-correct construction must beat it on real data; otherwise the precise model contributes nothing. The second baseline above tests exactly this. We expect the difference to appear mainly where clipping and the $2a$ variance term dominate. If it does not appear there, we will drop the claim.

Why not just use a generative separator like Flash-Split? It performs strongly: 32.06 dB against 28.60 dB for the previous flash-based method. It also runs on JPEGs. However, it needs iterative diffusion sampling, can hallucinate texture where the signal is weak, and was not evaluated under strong ambient light. The high-ambient FOR split directly tests whether its latent-space robustness already handles noise swamping. If it does, our contribution narrows to cost and latency, and we will measure and report both.

Why not just use single-image restoration? Software-only reflection removal plateaus at about 25 to 26 dB, per Flash-Split. Ambient normalization reaches only 22.116 dB on Ambient6K with PromptNorm. The information gained from the second frame is real, and the only open question is how to keep it usable in bright light.

Decisive checks

The first check tests the noise model. On the real calibration set, we compute the per-pixel variance of the difference image across repeated captures and compare it with the generator's predicted $2a + \alpha c$. If the fit is poor, the construction is wrong and the rest of the study is moot.

The second check tests the benefit to learning. For PSNR on the real outdoor bins, the ordering should be: ours, then independent-noise synthetic, then rendered, then raw subtraction. Neither synthetic-trained model should see real high-ambient data during training.

The third check tests the cost claim. We plot downstream PSNR against generation GPU-hours. The claim holds only if our generator matches or beats the rendered corpus at less than one percent of its generation cost.