Audited Triage: Routing Pixels Between Deferred and Forward Shading

Fast neural renderers for shiny objects mostly use deferred shading. They first paint each pixel's surface properties (depth, surface direction, color, roughness) into a 2D image called a G-buffer, and then run a small shading network once per pixel. This is why recent Gaussian-splatting methods render glossy objects at 45 to 200+ frames per second with sharp highlights. The trick rests on one assumption: each pixel sees exactly one opaque surface. Glass, water and thin translucent layers break that assumption, and the deferred methods themselves say so (RGS-DR, Stanford-ORB). Methods built for transparency, such as TransparentGS and an index-of-refraction-adaptive radiance field, instead shade along the ray before blending. That "forward" approach handles multiple layers but costs more and smears highlights on opaque surfaces. Real scenes contain both kinds of material, and today one has to pick a single pipeline for the whole image.

We propose to borrow the working structure of a journal editorial board. Most submissions are handled by a cheap first pass. Only uncertain cases are escalated to specialist reviewers. An editor spot-checks a random sample of routine decisions, so that the cheap pass cannot drift unnoticed. In rendering terms, a small per-pixel router reads cheap signals produced while the G-buffer is built. It sends most pixels to deferred shading and escalates a few to a forward multi-layer shader. A random audit renders a small fraction of non-escalated pixels through both paths. The measured gap between the two paths trains the router. The claim is that audited triage reaches the quality of forward shading on transparent regions at close to deferred-shading cost. It should also stay indistinguishable from pure deferred shading on fully opaque scenes.

Core Claim and What Is New

The deferred methods in the file (GI-GS, Ref-GS, RGS-DR, GlossyGS, GS-2DGS, GS-ROR2) all apply one shading path to every pixel. They differ in how that path models highlights and lighting, but none chooses a different path per pixel. The transparency methods also commit to a single path, the forward one, for the whole scene. What is new here is per-pixel choice of the shading pipeline under an explicit compute budget, with the choice learned from audited disagreement rather than set by a hand-tuned rule.

The audit matters because of a known failure in inverse rendering. Inverse rendering means recovering geometry and materials from photographs. When a pixel is routed to the cheap path, the optimizer does not leave the error alone. It "explains" refraction by painting distorted background into albedo, or by bending geometry, in the same way the file describes highlights being baked into fake bumps (GlossyGS). After that happens, the cheap path's own training loss no longer reveals that it is wrong.

Principal-agent analyses of multi-tier organizations make the same structural point. Delegated units optimize their local objective, and fidelity to the central goal decays across tiers unless there is vertical monitoring (Dong and Yang). We use that argument only as a design analogy, and the experiments test it.

Let $\Delta(p)$ be the audited error gap at pixel $p$: the photometric error of the deferred path minus that of the forward path on the training image. The router $r(p)\in[0,1]$ is trained to predict when escalation pays off under a budget $B$ on the escalated fraction:

$$\min_r \; \mathbb{E}_{p\in\mathcal{A}}\big[\ell(r(p),\, \mathbb{1}[\Delta(p)>\tau])\big] \quad \text{s.t.} \quad \mathbb{E}_p[r(p)] \le B .$$

Here $\mathcal{A}$ is the audited pixel set, $\ell$ is a classification loss, and $\tau$ is the smallest gap considered worth paying for.

Research Program

We build on a 2D-surfel deferred pipeline of the RGS-DR / Ref-GS type as the cheap tier. A forward multi-layer Gaussian shader in the style of TransparentGS serves as the specialist tier. The two share primitives, and the forward tier adds refraction parameters.

The router is a tiny network. Its inputs are features that come almost for free during rasterization: how spread out the blending weights are along the ray, residual transmittance, depth variance among contributing primitives, and normal disagreement among those primitives. All are computed per pixel.

Training proceeds in three stages. A short warm-up shades a downsampled image through both paths so that both have usable parameters. After that, each iteration renders escalated pixels through the forward path only, and renders a random audit set of a few percent of the remaining pixels through both paths. Photometric gradients flow through whichever path is assigned to a pixel. Audited pixels additionally produce router labels.

The "editorial" ingredients in this design are the tiering, the escalation budget and the random audit. We deliberately do not use language-model agents. Policy-as-Prompt shows that LLM adjudicators give different decisions on edge cases when only the prompt wording changes, at roughly 0.79 average accuracy. That kind of instability is unacceptable inside a gradient loop. Guo et al. find that a designated monitor or leader beats flat coordination. We take from them only the structural idea of a monitor that diagnoses the lower tier, and implement it numerically.

Baselines

The naive reference point is all-deferred shading: the same surfel pipeline with the router fixed to zero, equivalent to RGS-DR or Ref-GS. On opaque glossy data it is very strong. Ref-GS reports a normal mean angular error of 2.21° on Shiny Blender at over 45 FPS. GS-2DGS reports an average Chamfer distance of 0.0068 on Glossy Blender at 160 FPS. On opaque data we count as "beating" this baseline matching it within noise, with less than 1% of pixels escalated. On mixed scenes, beating it means clearly higher quality in the transparent regions.

The strongest competitor on the actual question is all-forward transparency rendering with TransparentGS. It is the method designed for the hard pixels. We found no reported result on Lacuna for TransparentGS, or for the refractive radiance field, on any benchmark in this file. We will therefore run TransparentGS ourselves on identical scenes, with the same GPU, the same iteration count and the same hyperparameter-search budget as our method. Beating it means equal PSNR/LPIPS in transmissive regions, higher quality on opaque glossy regions where forward blending smears highlights, and markedly higher frame rate.

The third baseline is a K-layer deferred G-buffer applied to every pixel, built with depth peeling with K of 2 to 4. This is the standard graphics answer to multi-surface pixels.

Two ablations isolate the key ingredient. The first is a fixed-threshold router on blending-weight spread, with no learning and no audit. The second is a learned router trained only on its input features, without audit labels. Their comparison with our method shows whether audits are what prevent the cheap path from silently absorbing errors. An oracle router that uses ground-truth material masks on synthetic scenes gives the upper bound.

All methods share data splits, primitive budgets and evaluation resolution. We report quality-versus-frame-rate curves rather than single points.

Why not just…?

Why not just use a fixed opacity or weight-spread threshold? This is the most serious threat. If transparent pixels always show multimodal blending weights, a hand-set threshold could match the learned, audited router. The audit argument predicts otherwise. Once the cheap path has baked refraction into albedo, a pixel's blending weights can collapse onto a single surface and look opaque. A threshold would then never flag the pixel again. The threshold ablation, measured on transparent-region error over training time, settles this. If the threshold matches us within noise, we will report that and drop the learned router.

Why not just use a multi-layer G-buffer everywhere? If transparent pixels are a large share of the image, or K-layer shading is cheap enough, triage buys nothing. We will measure the escalation fraction and the frame rate against the K-layer baseline across scenes with increasing glass coverage. Our claim covers only the regime where escalation stays small, and the crossover point is itself a reported result.

Why not just render everything forward, as transparency methods do? The file's opaque results argue against it. Forward blending across primitives with divergent normals blurs specular detail. Deferred pipelines are the ones reaching 2.21° normal error and 160+ FPS. The direct comparison on opaque glossy subregions of mixed scenes will show whether that gap persists inside a single scene.

Why not use an LLM "editorial board" to arbitrate G-buffer conflicts, as the intersection naively suggests? The prompt-sensitivity results in Policy-as-Prompt and the per-pixel, per-iteration call volume make this both unstable and impractical. The editorial structure is useful here as a design for routing and monitoring, not as literal agents.

Proposed Experiments

The first test is "do no harm." On Shiny Blender, Glossy Blender and the real-world Stanford-ORB, whose strongest reported relighting baseline is NVDiffRecMC at PSNR-H 24.43, the router should escalate almost nothing. Normal error, Chamfer distance, relighting quality and frame rate should then match the all-deferred baseline.

The second test targets mixed scenes. The file lists no mixed opaque–transparent benchmark with reported numbers. We will therefore build a synthetic suite by inserting glass and water objects of graded screen coverage, from 2% to 40%, into Shiny Blender scenes. Ground-truth material masks enable per-region metrics and the oracle router. We will also evaluate on the transparent-object captures released with TransparentGS, if those captures are available.

The decisive check concerns the audit. Without audits, transparent-region error should rise over training as the cheap path absorbs refraction, and the router's escalation rate on truly transparent pixels should fall. With audits, both should remain stable. We will also plot quality against escalation budget $B$ and audit rate, and report where audited triage stops beating the K-layer and all-forward baselines.