Who Gets Credit for the Shadow? Contribution-Aware Stage Transitions for Inverse Rendering

Inverse rendering recovers a 3D object's shape, surface material and lighting from photographs. When a system tries to estimate all three at once, it usually cheats. A shadow or a shiny highlight can be explained just as well by a dent in the surface or a darker paint, so the optimizer folds lighting effects into geometry and color. The fix the field has settled on is to optimize in stages. A simple diffuse (matte) model comes first, and shadows, subsurface scattering and specular reflection are switched on one at a time later. When each stage starts is set by hand-picked iteration counts. Those fixed schedules are a recognized weak point, and the first component to train has a structural head start: it can absorb effects that belong to components that do not exist yet.

That head start is a credit-assignment problem, and a field that has studied credit assignment for decades has a name for it. Studies of scientific authorship show that whoever is listed first absorbs recognition far beyond their share. Binary "equal contribution" tags fail to correct this. Fixing a misattribution afterward takes an explicit correction. We propose to treat the rendering components in a staged pipeline as co-authors of each image. Every few hundred iterations we measure each component's actual contribution to explaining held-out lighting conditions. Those measurements decide when the next component starts. They also decide when an earlier component that has taken over another component's role is forced to give it back.

Core Claim

Stage transitions in staged inverse rendering should follow a measured contribution signal rather than a fixed clock. The signal we propose has two parts. Each component's marginal share of the reduction in error on a held-out set of light directions tells us when the active components have stabilized. A leakage score tells us when one component is earning credit inside image regions that physically belong to another, for example the diffuse color term explaining pixels that occlusion tracing marks as shadowed. We expect such a controller to reduce "role leakage" and improve relighting compared with the hand-tuned schedule in SSD-GS. It does not have to beat the schedule on training-view fit.

The claim is falsifiable in a specific way. If a plain "start the next stage when training loss plateaus" trigger matches the controller, then the contribution measurement adds nothing and the authorship framing is decoration. We set up the experiments so that this outcome is visible.

Why the Authorship Literature Is the Right Lens, and Where It Stops

The authorship work supplies diagnoses, not algorithms. First Author Advantage: Citation Labeling in Research extracted citation keys from 12,611 arXiv source files. It found that 54.5% of keys contain the first author's surname, against 2% for a fifth author in five-author papers. This is the anchoring pattern we expect in a diffuse-first schedule, where the first-trained term becomes the default explanation for every bright or dark pixel.

Author credit-assignment schemas: A comparison and analysis compares fifteen fixed credit formulas, from equal counting to first-author-takes-all. It shows that the rank correlations between schemas are high, but individual standings shift substantially with arbitrary parameter choices. That is the same failure as a hand-picked iteration count. We replace the fixed formula with a measured one. Microsoft Academic Graph: When experts are not enough and A Review of Microsoft Academic Services for Science of Science Studies acknowledge that the graph splits credit equally among all coauthors as a simplification. This is the bibliometric analogue of joint optimization, in which every component has an equal claim on every gradient from the start.

The authorship dilemma: alphabetical or contribution? models authorship as a two-player game. Equal credit bounds free-riding (about 20% in their identical-agent, degree-2 setting). The paper also identifies rotating first authorship as a way to balance effort. We borrow both ideas. A "free-riding" component is one whose credit stays near zero while it is active, and we test rotation as a scheduling alternative. The long list of publisher errata restoring equal-contribution status, such as Correction to "Nonlinear Regression via Deep Negative Correlation Learning", motivates the third mechanism. When leakage is detected, the earlier component is not merely frozen. Its parameters in the affected region are partially reset so the correct component can claim the effect.

The analogy breaks down in one important place. Rendering components have physical ground truth, and authors do not. That is why the inverse-rendering side is where the idea can actually be tested. We make no claim about scientometrics.

Two papers that share the "equal contribution" phrase are not close prior work. Joint Equal Contribution of Global and Local Features for Image Annotation fixes equal loss weights between global and local features. We measure contributions and act on them rather than imposing equality. Shaping Parameter Contribution Patterns for Out-of-Distribution Detection analyzes parameter contributions inside a classifier. It is a precedent for contribution-based diagnostics, but it does not schedule optimization.

Research Program

The base system is SSD-GS (SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting). It represents objects with 3D Gaussian splats, which are many small semi-transparent ellipsoids rendered by projecting them onto the image. It splits appearance into diffuse, shadow, subsurface-scattering and specular terms, trained in that fixed order to avoid leakage. Because the final image is a composition of these four terms, any subset of them can be rendered cheaply by switching the others off. That makes exact contribution measurement affordable.

From the training lights we hold out a small probe set of one-light-at-a-time images. These images are never used for gradients. Let $C$ be the set of active components, $n=|C|$, and $E(S)$ the probe-set photometric error when rendering with only the components in $S \subseteq C$. The credit of component $c$ is its Shapley value, the average marginal error reduction it brings over all orders of adding components:

$$\phi_c = \sum_{S \subseteq C\setminus\{c\}} \frac{|S|!\,(n-|S|-1)!}{n!}\,\big[E(S) - E(S\cup\{c\})\big].$$

With at most four components this takes 16 renders of a small probe batch, which we amortize by evaluating every $K$ iterations.

Leakage is the share of component $c$'s per-pixel credit that falls inside a mask $M_d$ where physics says component $d$ should dominate:

$$\Lambda_{c\to d} = \frac{\sum_{p \in M_d} \phi_c(p)}{\sum_{p} \phi_c(p)}.$$

Here $\phi_c(p)$ is the same quantity computed per pixel. Masks are taken from quantities the pipeline already computes. Occlusion tracing gives shadow masks. Regions near the mirror-reflection direction with high specular lobe response give specular masks.

The controller applies three rules. A new component starts when the credits of the active set change by less than a tolerance over a window. Training loss alone does not trigger a transition. If $\Lambda_{c\to d}$ rises above a threshold after $d$ has started, the controller runs a correction: $c$'s parameters inside $M_d$ are pulled back toward their pre-$d$ values, and $c$ gets a short extra training phase. A component whose credit stays negligible for a long window is flagged as free-riding and has its learning rate reduced. All tolerances are tuned once on one synthetic scene and then held fixed across every dataset. To test generality we port the same controller to the two-stage diffuse-to-specular pipeline of 3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting. With $n=2$, the Shapley credit reduces to an average of two orderings.

Baselines

The naive reference point is joint optimization: all four SSD-GS components are active from iteration zero, with no schedule. This is the setting staged methods were designed to escape, and it gives the floor for leakage.

The strongest existing method is SSD-GS with its published fixed progression of diffuse, then shadow, then scattering, then specular. On the real NRHints Pixiu scene it reports 31.12 dB PSNR, against 30.11 dB for the previous best (SSD-GS). We rerun it from its released configuration on identical splits and identical total iterations. We also give it a fair tuning budget: a grid search over its stage-transition iterations, using as many full training runs as our controller consumed during its one-scene tuning. The comparison is then against a well-tuned fixed schedule rather than a default one. Beating it means higher relighting PSNR and lower LPIPS on held-out lights averaged across NRHints scenes, plus lower ground-truth component error on the synthetic sets, at matched wall-clock time including the cost of computing credits.

For the two-stage port, the reference is 3iGS's fixed diffuse-then-specular schedule, which reports an average of 33.64 dB PSNR on NeRF Synthetic versus 33.30 for vanilla 3DGS (3iGS). We found no reported result on Lacuna for 3iGS on Shiny Blender, GS³ or SSS-GS, so on those sets we report our own reruns only.

Three ablations isolate the key ingredient. The first is a loss-plateau trigger, which starts the next stage when the training loss plateaus and uses no credit measurement. The second is leave-one-out credit in place of Shapley credit. The third is the full controller with the correction step removed, which tests whether reopening earlier components matters or whether better-timed starts are enough.

Why not just…?

Why not just tune the fixed schedule harder? This is the most direct competitor, and it is included as the grid-searched SSD-GS baseline at a matched training budget. Our argument is that the best transition point differs from object to object. A candle with strong subsurface scattering needs a different schedule than a matte figurine. The test is whether a single controller with fixed tolerances beats per-dataset tuned schedules on scenes it was not tuned on. If the tuned fixed schedule wins on held-out scenes, our claim fails.

Why not just start each stage when the training loss plateaus? This is a real threat. Loss plateaus are cheap and already signal convergence. The difference is that a plateau on training views cannot tell a converged diffuse term apart from one that has stopped improving because it has already absorbed the shadows. Credit on held-out lights and the leakage score can tell them apart. The plateau-trigger ablation decides this directly. If it matches the controller on relighting and on component error, we drop the credit machinery and report that result.

Why not just anneal the loss weights, as DeLiRa does? DeLiRa: Self-Supervised Depth, Light, and Radiance Fields emphasizes Lambertian consistency early and relaxes it later. That is a soft schedule, but it is still a clock. It also has no way to undo leakage once it has happened, and the correction step is exactly that mechanism. We run an annealed-weight variant of SSD-GS under the same budget. We expect the no-correction ablation to perform about as well as annealing, and the full controller to differ from both mainly in shadow-region error.

Proposed Experiments

The primary benchmark is the real NRHints one-light-at-a-time data (Cat, Pikachu, Pixiu), where we evaluate relighting under held-out light positions. The synthetic GS³ scenes (Lego, Drums) and SSS-GS scenes (Bunny, Dragon, Candle) are re-rendered with each component isolated wherever their scene assets allow. That gives per-component ground truth for diffuse, shadow and specular images. We measure leakage against it directly, and on GS³ we also measure normal error. For the generality test we use NeRF Synthetic and Shiny Blender with the 3iGS port.

The first decisive check validates the measurement before trusting the controller. Across many differently timed fixed schedules on the synthetic scenes, the leakage score measured during training should rank runs in the same order as their final ground-truth component error. If it does not, the controller is acting on noise, and we stop there.

The second check is the head-to-head against tuned SSD-GS and the plateau trigger on NRHints held-out lights, with the controller's tolerances frozen after tuning on a single GS³ scene.

The third check is a targeted stress test. We start the shadow component deliberately late, so the diffuse term has maximal opportunity to absorb shadows. We then ask whether the correction step recovers shadow-region error close to the well-scheduled level, where freezing alone would leave it baked in.

We will also report cost, both the fraction of wall-clock time spent on credit evaluation and the sensitivity to the evaluation interval $K$. A controller that wins only by spending extra compute does not count as a win.