When a projector shines onto a patterned wall, a wooden table or a curved piece of fabric, the surface's own colors mix with the projected light and the image turns muddy. Projector compensation, meaning computing a pre-distorted input so the image looks correct once it reflects off that surface, works well today. The catch is that the best neural methods must be retrained for every room: they photograph hundreds of projected image pairs and optimize a network on site. We propose a different route. We train a single compensation network entirely on simulated projector-camera setups in which the projector, camera and surface properties are heavily randomized. At deployment, the network is given a handful of "probe" photographs of the new setup (flat black, white and color fields plus two textured patterns, captured in seconds) as extra input. It then compensates new content immediately, with no gradient steps on site.
The bet is that the part of the synthetic-to-real gap that matters most for compensation is narrow and observable. That gap is the performance drop when a model trained on rendered data meets real photographs. We think it is dominated by per-device radiometric response, gamut limits, defocus blur and ambient light, and each of these is directly visible in a few probe images. If that is right, the network does not need to be fit to the room. It only needs to be told about it.
Core Claim
A network conditioned on about eight probe captures of a new setup, and trained only on randomized physically-based simulation, can match per-setup-trained neural compensation on real benchmarks. It should also beat a synthetically pretrained network fine-tuned on those same eight captures. The comparison between conditioning and fine-tuning at equal capture budget is the new and decisive part.
Existing work uses simulation only as initialization, or needs on-site optimization. CompenHR pretrains its photometric subnet on 100 Blender setups and then still trains on 500 real pairs. LAPIG pretrains both its compensation network and its project-and-capture simulation network on 10 Blender setups, then tunes them on real pairs. DPCS fits an inverse path-tracing model from 5–15 real pairs but needs seconds per frame. None of them tests whether a simulation-trained model can be steered to a new physical setup by conditioning alone. None of them measures which simulated factors cause transfer to fail.
Research Program
The simulator is built on the physically based forward model that DPCS uses. That model is Mitsuba 3 path tracing with a Disney-style surface reflectance model, a projector response function and a camera response function (the curves mapping input values to emitted light and captured light to pixel values), and exposure. We add the explicit hardware terms from GS-ProCams: a projector defocus blur kernel, gamma and gain. We then randomize, per synthetic setup, each of the following: surface texture and geometry; roughness, including a tail of glossy materials; the projector and camera response curves and color gamut; blur; ambient and indirect light; vignetting; and camera noise, including clipping. Rendering is offline, so its cost is paid once.
The network follows the CompenNet family. CompenNet already conditions a UNet on a surface image through an autoencoder branch. We widen that branch to take the full probe stack, aligned to the target view. For each pixel, the black and white probes directly show the achievable light range. This matters because projectors cannot subtract light (Fujii et al.; FRESCO), so the network sees where clipping is unavoidable instead of having to guess.
Geometric correspondence comes from standard Gray-code structured light, a short sequence of stripe patterns. We deliberately keep this step classical, so the study isolates the photometric sim-to-real question. The same correspondence is given to every baseline.
Training follows the loss design of LAPIG. A differentiable simulation of projection and capture renders the network's output, and the loss compares that rendered capture to the target. It adds the saturation penalties LAPIG introduced. Written out, with $G$ the compensation network, $y$ the desired image, $P$ the probe stack, and $F_\theta$ the simulated projection and capture under setup parameters $\theta$, we minimize
$$ \mathbb{E}_{\theta, y}\big[\,\ell\big(F_\theta(G(y; P_\theta)),\, y\big)\big], $$
where $\ell$ is a pixel loss plus structural similarity, and $P_\theta$ are the probes rendered for setup $\theta$. Because $\theta$ is resampled for every setup, $G$ must read the setup from $P_\theta$ rather than memorize it.
The second output is a diagnosis of the gap. We retrain with each randomization factor switched off in turn, or with the probe input removed. We then report how much real-world accuracy each factor accounts for, alongside the synthetic-test versus real-test difference. Real-world accuracy is measured by PSNR (peak signal-to-noise ratio, in dB, higher is better). This follows the "shortcut" lesson of ASM: clean simulated sensors let networks rely on the absence of noise, and noise that realistically mimics the sensor removes that crutch.
Baselines
The naive reference points are uncompensated projection and classical interpolation. On the CompenNet++ full-compensation benchmark, uncompensated projection reports 9.60 dB and a thin-plate-spline method reports about 18.03 dB. On the CompenNet benchmark, the spline method reports 19.23 dB. A result above the spline numbers is the bare minimum.
The ceiling reference is per-setup supervised training on the full real training set. CompenNet reports 21.80 dB, CompenNet++ reports 21.11 dB, and CompenHR reports 20.95 dB on its 24-setup high-resolution set, against 20.55 dB for its CmpSt(HR) baseline. We rerun these with each paper's full real data. Coming within 0.5 dB of them with eight probes and zero on-site training would count as success.
The strongest competitor at equal capture budget is Blender-style synthetic pretraining followed by fine-tuning, which is the CompenHR recipe. We pretrain the same architecture on our simulator without probe conditioning. We then fine-tune it on exactly the eight probe captures, where the two textured probes serve as paired examples, with a tuned learning rate and step budget chosen on held-out setups. No reported result was found on Lacuna for CompenHR or CompenNet++ fine-tuned on fewer than 32 real pairs, so we produce that curve ourselves at 8, 16, 32 and 64 captures. Beating it at 8 and matching it at 32 would show that conditioning replaces fine-tuning.
The strongest few-shot method overall is DPCS. It reports about 31.7 dB with 15 samples on its own dataset, against about 29.7 dB for DeProCams. We run DPCS on that dataset with 8 and 15 captures and report accuracy together with wall-clock time to first compensated frame.
The key ablations remove the probe branch (pure domain randomization) and remove the noise and response randomization (clean simulation).
Why not just…?
Why not just fine-tune a Blender-pretrained network on a few captures? This is the main threat, and it is run as a baseline at matched budgets. Our expectation is that gradient fine-tuning on eight images overfits the textured probes and ignores the flat-field ones. If it matches our method, the paper's conclusion becomes "simulation pretraining plus tiny fine-tuning suffices," which is itself a finding no paper in the file reports.
Why not just fit DPCS and distill it into a fast network? DPCS's physics fit is excellent with 15 samples. Rendering thousands of training pairs from it and training a CompenNet-style student would give real-time inference. The cost is minutes to hours of rendering and training per setup, and every movement of the projector or surface resets that cost, which is the retraining bottleneck noted for CompenNet++ and DeProCams. We run this distillation on the DPCS dataset. If it beats us on accuracy, the honest claim narrows to time-to-deploy, and we report both.
Why not just randomize the simulator heavily and skip the probes? Domain randomization forces invariance (Understanding Domain Randomization). Compensation, however, must invert the specific surface and the specific device rather than ignore them, and a network cannot know a projector's gamut or a wall's albedo without seeing it. The probe-removal ablation measures this directly.
Why not refine renders with unpaired real photos, as S2R does for screens? S2R cut watermark bit error from 3.3% to 2.1% by adversarially refining simulated screen-camera distortions. Compensation, though, is judged on exact radiometry, where a GAN refiner can shift colors while looking plausible. We test it as an add-on to our training data, not as the core method.
Proposed Experiments
Training uses only simulated setups. Evaluation covers the real CompenNet, CompenNet++, CompenHR (including its 5 specular surfaces) and DPCS datasets, reporting PSNR, SSIM and RMSE, time to first frame, and frames per second.
The first decisive check is the capture-budget curve against fine-tuning described above. The second is the factor-ablation table, which should show that response and noise randomization together account for most of the real-world gain. If surface-appearance realism turns out to dominate instead, the core claim is wrong. The third is a failure analysis on the specular surfaces, where we expect flat-field probes to underdetermine view-dependent highlights. We report that case separately rather than averaging it away.
References
- CompenHR: Efficient Full Compensation for High-resolution Projector
- LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization
- DPCS: Path Tracing-Based Differentiable Projector-Camera Systems
- GS-ProCams: Gaussian Splatting-Based Projector-Camera Systems
- End-To-End Projector Photometric Compensation.
- A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments
- FRESCO: Fast Radiometric Egocentric Screen Compensation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- CompenNet++: End-to-End Full Projector Compensation
- DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
- Understanding Domain Randomization for Sim-to-real Transfer
- Sim-to-Real: An Unsupervised Noise Layer for Screen-Camera Watermarking Robustness