Pricing Light-Transport Shortcuts: A Counterfactual Benchmark for Fast Inverse Rendering

Fast 3D reconstruction methods now recover a scene's geometry, surface color, and shininess well enough to relight it, and some run at more than 100 frames per second. They get that speed by simplifying how light is simulated. They drop light that bounces between surfaces, treat every bounce as matte, trace reflections only within what the camera currently sees, or bake shadows into a coarse grid. Nobody can say what each shortcut costs. When a method recovers a wrong surface color, we cannot tell whether the cause is its shadow approximation, its missing specular bounces, or its geometry. When it runs fast, we cannot tell how many milliseconds a better transport model would add. Published comparisons mix different scene representations, training schedules, GPUs, and datasets, so the field has no price list for its approximations.

We propose to build one. The core device is counterfactual ground truth: synthetic scenes rendered several times with a physically based path tracer, each version with a specific class of light paths switched on or off. The classes are direct light only, cast shadows, diffuse interreflection, specular interreflection, and light from outside the camera's view. A method that ignores specular interreflection can then be scored on the version of the scene where that path class truly does not exist. The gap between that score and its score on the full scene is the error its shortcut causes. We pair this with a shared rendering backbone into which each published transport approximation plugs as a swappable module, timed on fixed hardware. Every approximation then gets two numbers: how much physical accuracy it gives up, and how many milliseconds it saves.

Core Claim

The error a fast inverse-rendering method makes when it recovers material properties (albedo, the surface's base color, plus roughness and normals) can be split into a transport penalty, caused by light paths the method's simulator cannot represent, and a residual caused by everything else. Measured on a common backbone, these penalties will rank existing approximations differently from the rankings implied by novel-view image quality. Some of the fastest approximations will turn out to be cheap for novel-view metrics but expensive for albedo and relighting. Concretely, for a method $m$ and error metric $E$ (for example albedo PSNR or normal angular error), we define

$$\Delta_T(m) = E\big(m;\ \text{full transport}\big) - E\big(m;\ \text{transport restricted to what } m \text{ models}\big),$$

where the second term evaluates $m$ on a counterfactual rendering from which the path classes $m$ omits have been removed. We then define the marginal price of adding transport class $c$ to a backbone as $\rho_c = \Delta E_c / \Delta t_c$, the reduction in error per added millisecond of frame time $t$. The first quantity separates "the simulator is too simple" from "the optimizer or representation failed." The second turns the latency-fidelity trade-off into a comparable number, in the spirit of the decoupled accuracy–robustness axes used in VOT2019.

Why This Is Not Already Done

The benchmark side of the literature motivates the proposal but stops short of it. OpenIllumination captures 64 real objects under 142 one-light-at-a-time LED patterns. It shows that handheld capture corrupts inverse-rendering evaluation, and that fidelity falls from up to 35.77 PSNR on easy objects to about 22 PSNR on translucent ones. It measures neither latency nor which transport term causes the drop. The AIM 2024 Sparse Neural Rendering Challenge supplies the methodology we reuse: hidden test ground truth on the Codabench server with fixed tracks. It evaluates only view-synthesis color under baked lighting.

The method side supplies the modules we will swap and also constrains the design, because each paper evaluates its own approximation on its own terms. GS-IR bakes visibility into spherical-harmonic grids. RTR-GS replaces indirect light with one learnable diffuse scalar plus baked occlusion and reports 96.4 and 130.9 FPS. 3D Gaussian Inverse Rendering with Approximated Global Illumination uses screen-space reflections, meaning rays traced only through the depth and normal images the camera already rendered. It reports about 1 ms of overhead and about 37 FPS at 960×540 on an RTX 4090. GI-GS adds a world-space cubemap so bounces can reach off-screen geometry, but it is diffuse-only and reports no rendering-latency profile. RGS-DR uses distant-environment lighting with an image-space residual and admits failure on near-field interreflection scenes such as "coffee" and "teapot."

Slower methods show that explicit indirect modeling matters for albedo, but they give no timings. Modeling Indirect Illumination reports 25.25 versus 22.96 aligned albedo PSNR over NeRFactor on four CAD scenes. Physics-based Indirect Illumination traces rays against an implicit surface. TensoIR ray-marches its radiance field for secondary bounces, and Baking Relightable NeRF distills it into a 12–70 FPS student that inherits the teacher's biases.

None of these works uses counterfactual transport ground truth, and none compares approximations on a shared backbone with controlled timing. That combination is the contribution.

Research Program

Counterfactual scene suite. We will author roughly 40 synthetic object and small-room scenes in a physically based path tracer. They will span matte, glossy, metallic, and translucent materials, chosen so that each transport class matters in some scenes and not in others. Metals are included deliberately, because the fixed dielectric Fresnel term ($F_0 = 0.04$ in TensoIR, $0.02$ in Modeling Indirect Illumination) should incur a measurable penalty there. Each scene is rendered at 1600×1200 under training and relighting illuminations, in five versions: direct unshadowed, direct shadowed, plus diffuse interreflection, plus specular interreflection, and full transport. Path-class filtering is done at the light-path level inside the path tracer, not by compositing images, so every version is physically consistent. Ground-truth albedo, roughness, metallicity, and normals are shared across versions, so any change in a method's recovered materials comes only from transport.

Frustum-truncation trajectories. A subset of camera paths is designed so that a bright emitter or strong reflector sweeps out of view while it keeps illuminating the visible surface. We score temporal stability with the frame-to-frame change in rendered indirect radiance relative to ground truth. Screen-space methods are expected to fail on exactly these paths.

Shared backbone with swappable transport modules. We fix one deferred-shading backbone: 2D Gaussian surfels rasterized into per-pixel buffers of depth, normal, and material, following RGS-DR. Surfels are chosen for their well-defined normals, and we keep one training schedule. Onto it we port six transport modules. The first is split-sum environment lighting with no occlusion. The second is spherical-harmonic visibility grids, as in GS-IR. The third is baked occlusion with a learnable diffuse indirect scalar, as in RTR-GS. The fourth is screen-space one-bounce tracing. The fifth is buffer tracing plus a world cubemap, as in GI-GS. The sixth is a reference that traces secondary rays against the surfels themselves. Each is timed with GPU timestamp queries over a fixed 300-frame trajectory on an RTX 4090 at 960×540 and 1600×1200, and we report median and 95th-percentile frame time plus the per-module share.

Real-data validation. Counterfactual versions cannot be captured physically. We therefore test whether penalties measured on synthetic data predict each module's relighting-error ranking on OpenIllumination, stratified by its material categories. The test set stays hidden behind a Codabench server, as in AIM 2024.

Baselines

The naive reference point is vanilla Gaussian splatting with view-dependent color baked into spherical harmonics and no transport model at all. It is the fastest option; the file cites 300 FPS for FlashAvatar. It cannot relight, so it anchors the latency axis and the view-synthesis ceiling, and it shows how far view-synthesis PSNR can drift from material accuracy. It is trained on the same scenes and views with the same iteration budget as the backbone.

The strongest existing competitors are RTR-GS and GI-GS. RTR-GS pairs the best view-synthesis quality in the file, 41.39 PSNR on the TensoIR synthetic set with its hybrid branch (RTR-GS), with 130.9 FPS for its physically based branch. GI-GS is the only fast method in the file whose indirect bounces can reach off-screen geometry; it reports 36.75 PSNR on the same set against 35.33 for GS-IR (GI-GS). For albedo and relighting accuracy on the TensoIR set we found no reported result on Lacuna for either method, so we will measure these ourselves. Both methods are run twice. The first run uses their released code and published settings, to reproduce the reported numbers within noise; if we cannot reproduce them, we report that. The second run uses their transport module ported onto the shared backbone, with the same per-scene hyperparameter search budget of 20 trials for every module. A module "beats" RTR-GS in our framing if it sits on the latency–albedo Pareto front where RTR-GS does not, meaning lower albedo and relighting error at equal or lower 95th-percentile frame time. For the screen-space module the latency reference is the reported ~1 ms overhead and ~37 FPS at 960×540 (3D Gaussian Inverse Rendering with Approximated Global Illumination). Baking Relightable NeRF, at 12–70 FPS on an A100, is included as the distillation reference and run on the 4090 for parity.

The key ablation isolates the counterfactual ingredient. We compute every conclusion twice: once with the full counterfactual suite, and once with only full-transport ground truth, which is how existing benchmarks such as the TensoIR set work. If the attribution of error to transport classes and the Pareto ordering are unchanged, the counterfactual renderings add nothing and the proposal reduces to a timing study.

Why not just…?

Why not just add an FPS column to OpenIllumination or the TensoIR benchmark? This is the cheapest option, and for pure ranking it may partly work. It tells us which complete method is faster and more accurate. It cannot tell us why a method's albedo is wrong, because on full-transport ground truth a missing specular bounce and a geometry floater produce the same symptom: shading baked into color. OpenIllumination's own 35.77-to-22 PSNR drop across materials is exactly this kind of unattributed gap. The key ablation above tests directly whether FPS-plus-full-GT reaches the same conclusions as the counterfactual suite.

Why not just compare published methods end to end? Published numbers conflate transport with representation: 3D Gaussians versus surfels versus tensor fields versus SDFs. They also conflate transport with training time (0.5 h for RTR-GS against 28–33 minutes for GI-GS) and with hardware (A100 versus 4090). This is a real threat if the end-to-end ranking and the module-swap ranking agree, in which case the backbone adds cost without insight. We run both and report their rank correlation. If it exceeds 0.9 on every metric, we will say the shared backbone was unnecessary for ranking and keep it only for per-millisecond pricing.

Why not just distill a full path tracer into a fast student and skip approximations entirely? Baking Relightable NeRF shows that the student inherits its teacher's material errors and cannot adapt to changed geometry. The approach also still needs a teacher with correct materials, which is the very quantity under study. We include it as a baseline; if it dominates the Pareto front on the counterfactual suite, that finding is itself decisive for the field.

Proposed Experiments

The first check is whether transport penalties are real and large. On matte scenes, the diffuse-interreflection penalty for RTR-GS's scalar module should be small. On concave glossy and metallic scenes, it should grow substantially. The effect should be traceable to the scenes where RGS-DR reports failures. If penalties stay small everywhere, the transport-shortcut concern in the literature is overstated, and we will report that.

The second check is whether view-synthesis rankings and albedo/relighting rankings diverge across modules. We expect them to, following the evidence in Modeling Indirect Illumination that view quality hides decomposition errors.

The third check is the frustum trajectories. The screen-space module should show large temporal instability when an emitter leaves the view, and the cubemap module should remove most of it for some added frame time. We will quantify that added time as $\rho_c$.

The fourth and decisive external check is transfer to real data. Penalties measured on synthetic scenes should predict the ordering of modules' relighting errors on OpenIllumination within each material category, with Spearman correlation above 0.6. Failure would mean the synthetic price list does not transfer to real captures, which limits the benchmark's value, and we would publish it as such.

All scenes, counterfactual renderings, the shared backbone, the timing harness, and the hidden-split server will be released.