Normal-Space Priors for Shiny Moving Objects Seen by Few Cameras

When a handful of fixed cameras film a moving glossy object, such as a metal kettle being turned or a car passing a storefront, current reconstruction methods fail. Methods built for few cameras assume every surface looks the same from every angle. They therefore read a highlight that slides across the kettle as a change in geometry and carve false dents or floating blobs to explain it. Methods built for shiny moving objects avoid that mistake, but only because they rely on one camera sweeping densely through the scene. Our proposal rests on one observation. A surface's orientation (its normal, the direction it faces) does not depend on where the viewer stands, while its color does. A generative prior that predicts normals instead of colors therefore gives each camera geometric evidence that should agree across cameras even on a mirror. Disagreement between that evidence and the photographs is then itself informative: a residual that a reflection model can explain is a highlight, and a residual it cannot explain is a geometry error.

Concretely, we run a pretrained video normal estimator independently on each sparse camera's video. That camera is dense in time even though the rig is sparse in space. We rotate its predictions into a shared world frame and fuse them across cameras with a confidence weight based on agreement. The fused normals then serve two purposes inside a dynamic reflective splatting model. They regularize geometry, and they drive a reflection-aware consistency test that replaces the photometric checks used by sparse dynamic methods, which wrongly discard shiny surface points.

Core Claim

In sparse multi-camera dynamic capture of non-Lambertian objects, a generative prior should act on normals, not RGB. Its output should also be gated by cross-camera agreement, not by feature-matching attention. With both choices, the method should recover correct geometry and sharp moving highlights where both photometric-consistency methods and RGB-diffusion repair methods fail. The new element is not "add a diffusion prior to 4D splatting." It is the use of a view-independent quantity as the channel through which the prior enters, combined with the reflection-explained residual as the test that separates appearance changes from geometry errors.

Where this sits in prior work

Three groups of work motivate the gap. First, dynamic specular methods handle moving reflections but need dense monocular video. NeRF-DS conditions color on observation-space position and normals. SpectroMotion adds a deformable environment map and a static, then dynamic, then specular training schedule. ReflFlow uses flat 2D Gaussian disks so that normals are explicit, ray-traces near-field reflections, and already regularizes temporally with the video normal prior NormalCrafter. ReflFlow is the closest existing work. It uses that prior as a per-video smoothness term, relies on dense optical flow, and never confronts sparse cameras. We use the same prior across cameras as the geometric anchor and as the arbiter of consistency.

Second, sparse dynamic methods handle few cameras but assume matte appearance. GC-4DGS filters multi-view stereo points with a space-time photometric consistency check and adds depth-ranking losses. Shiny points fail that check and are discarded. Splatography handles reflective and transparent textures with 4 to 10 cameras, but its authors name the entanglement of view-dependent color with point motion as an unsolved failure. That admission is direct evidence for our gap.

Third, sparse static methods with generative priors fix floaters but confound highlights with texture. GSFixer restores renders with a video diffusion model and distills the result back. OracleGS proposes diffusion views and validates them with cross-view attention from a multi-view transformer. G4Splat anchors scale with planes. All three operate on static scenes and RGB. OracleGS's attention-based validator is a design we deliberately avoid, because moving glints destroy feature correspondence and would cause valid specular content to be rejected.

Two papers supply technique, which we reuse. GlossyGS contributes normal prefiltering before shading, which keeps highlights from being blurred by blending. ReflFlow contributes its 2D-disk ray-tracing renderer.

One paper constrains the design, and we address it directly. NormalCrafter reports a 30.7° mean angular error on Sintel. ReflFlow notes that a few degrees of normal error sends reflection rays to the wrong part of the environment. The prior therefore cannot be trusted to supply fine normals. In our design it supplies coarse orientation and a confidence signal, and the photometric reflection fit refines the normals locally.

Research Program

The representation is ReflFlow-style 2D Gaussian disks in a canonical space. A hex-plane deformation field moves them over time. A split shading model renders each disk: a diffuse term plus a specular term that traces reflection rays into an environment represented by Gaussians, with GlossyGS-style prefiltering of normals.

The first component fuses normals across cameras. For each camera $c$ and frame $t$, NormalCrafter produces a normal map. We rotate it into world coordinates using the known calibration and splat it onto the current geometry. At a surface point $x$, the prior normals $n_c(x,t)$ from the cameras that see it are combined into a fused normal $\bar n(x,t)$, with confidence

$$w(x,t) = \exp\!\Big(-\tfrac{1}{\sigma^2}\,\tfrac{1}{|C_x|}\textstyle\sum_{c\in C_x} \angle(n_c, \bar n)^2\Big),$$

Here $C_x$ is the set of cameras observing $x$, $\angle$ is the angle between two unit vectors, and $\sigma$ is an angular tolerance. Points seen by a single camera receive a fixed low weight. A weighted normal loss $w\,(1-\langle n_\theta,\bar n\rangle)$ pulls the model's normals $n_\theta$ toward the fused prior. Agreement between independently computed per-camera predictions is our substitute for an attention oracle, and it stays valid on shiny surfaces because normals do not change with viewpoint.

The second component is the reflection-aware consistency check. GC-4DGS rejects a candidate point when its color disagrees across cameras. We instead fit the diffuse and specular terms at the candidate using the fused normal. We keep the point when the residual after the reflection fit is small relative to the residual of a diffuse-only fit:

$$\rho(x,t) = \frac{\|I - \hat I_{\text{diff+spec}}\|}{\|I - \hat I_{\text{diff}}\|} < \tau .$$

Here $I$ is the observed color across cameras, $\hat I$ are the model fits, and $\tau$ is a threshold. A large disagreement in color that the specular term explains marks a valid shiny point. A residual that persists after the reflection fit marks a geometry error or a floater, and the point is pruned. Densification follows the same signal, so that transient highlights stop spawning floating Gaussians.

The third component is a limited completion step. For regions that only one camera sees, we let the low-weight prior normal shape the surface and leave appearance there to the diffuse term. We deliberately avoid RGB hallucination, so that invented highlights never enter training.

Training follows SpectroMotion's staged schedule. Static geometry comes first under the normal prior. Deformation is added next, and the specular term is introduced last, after the consistency check has stabilized the geometry.

Baselines

The naive reference point is 4D Gaussian splatting trained on the same 3 or 4 cameras with no priors. It shows how much of the problem the sparse setting itself creates.

The strongest sparse dynamic competitor is GC-4DGS, which is built for exactly this camera count. We run it with its released consistency checking and depth losses on identical camera subsets and frame ranges. It gets the same GPU budget and the same number of hyperparameter trials we use, and its monocular depth source is matched to ours. On N3DV with 3 input views it reports a PSNR gain of +1.58 dB over 4DGS. The file gives no absolute number, so we will reproduce both models ourselves and report the gap. Beating it means higher PSNR and lower LPIPS on the test cameras, measured on shiny-object masks rather than only full frames, together with lower normal error on synthetic data.

Splatography is the competitor that already claims to handle reflective and transparent textures under sparse cameras. It reports up to 3 PSNR higher than earlier methods on 3D scenes. We run it on DyNeRF's 4-camera setting, where it was tuned. Beating it there with a margin concentrated on reflective regions would show that we resolve its acknowledged failure mode.

ReflFlow is the strongest dynamic specular method. On NeRF-DS it reports +0.74 dB over SpectroMotion, which itself reaches 24.17 mean PSNR. We extend ReflFlow naively to multi-camera input by pooling all cameras' frames, keeping its NormalCrafter temporal loss per camera. This baseline isolates whether cross-camera fusion and the new consistency check matter, as opposed to simply running a good specular model on more data.

Two ablations isolate the key ingredients. In the first, we keep our full model but replace the reflection-aware check with GC-4DGS's photometric check. In the second, we keep the normal prior but set the agreement confidence $w$ to 1 everywhere.

Why not just…?

Why not just raise the spherical-harmonic order in a sparse 4D model? Spherical harmonics tie view-dependent color to a fixed frame. When an object rotates, its highlight should move in a way they cannot express without distorting geometry. SpectroMotion shows this on NeRF-DS: standard-harmonic Deformable 3DGS reaches 23.43 PSNR against 24.17 with physically based shading. That comparison uses dense video, however, and with few cameras the extra harmonic parameters are more likely to overfit. We include GC-4DGS with higher-order harmonics as a run, and we expect it to look acceptable on training cameras and degrade on held-out ones.

Why not just add NormalCrafter's normals as a loss to GC-4DGS? This is the real threat, because it captures much of the idea at low cost. It still leaves GC-4DGS's photometric check discarding shiny points before the normal loss can act, and it trusts a prior with roughly 30° errors on hard video. Our first ablation (normal loss plus the old check) and our second (no agreement weighting) are exactly this alternative. If they match the full model, the contribution reduces to a recommendation, and we will report that outcome.

Why not just remove highlights in 2D and train a matte model? Highlight removal applied frame by frame flickers. The Split4D and Splatography analyses describe how such 2D inconsistencies project into 3D as floaters and edge artifacts. Removal also throws away the reflection, which is part of what must be rendered at test views. We will run an off-the-shelf removal followed by GC-4DGS and measure both geometry error and reflective-region LPIPS. We expect the geometry to be competitive and the renders to look plausibly matte but wrong.

Why not just repair renders with an RGB video diffusion model as GSFixer does? RGB repair learns to treat a sharp glint as an artifact or to invent one, because it sees color, not material. It also costs 4.32 s per frame across 50 denoising steps. We will apply GSFixer-style repair to the GC-4DGS output and measure whether highlight positions at test views match ground truth on synthetic data.

Proposed Experiments

No existing benchmark combines sparse cameras, motion, and ground-truth normals for shiny objects. We will therefore render a synthetic set: animated Shiny Blender-style objects (rigidly rotating, articulated, and gently deforming) captured by 3 to 4 cameras, with ground-truth normals and held-out cameras. This set supports normal mean angular error and Chamfer distance alongside PSNR, SSIM, and LPIPS. Real-data evaluation uses DyNeRF and N3DV subsampled to 3 or 4 cameras, following GC-4DGS and Splatography. There we report full-frame metrics and metrics restricted to manually masked reflective regions.

NeRF-DS serves as a dense-monocular check. Our system must not fall below SpectroMotion's 24.17 when the multi-camera components have nothing to fuse. Static Shiny Blender, where GlossyGS reports 25.72 PSNR, checks that the shading model has not been weakened.

The decisive experiment sweeps highlight strength on the synthetic set, from matte to mirror, at 3 cameras. If our central claim holds, the gap between our method and GC-4DGS should widen as surfaces become shinier, and the gap to the naive normal-loss variant should appear mainly on reflective regions. A flat gap would mean the reflection-aware check adds nothing.

A second check correlates the agreement weight $w$ with the true normal error on held-out data. If the correlation is weak, cross-camera agreement is not a usable oracle and the design must change.

The third check counts floating Gaussians spawned by transient highlights. It tests directly whether densification gated on the reflection-explained residual stops this failure.