When a person or a car moves through a scene filmed by only two to four fixed cameras, its cast shadow on the ground is usually treated as a nuisance. Background-subtraction systems merge the shadow into the object and produce a distorted blob. 3D reconstruction methods behave in one of two bad ways. They paint the shadow onto the floor as a time-varying texture, or they explain the dark patch with spurious geometry: semi-transparent "floaters" hanging between the camera and the floor, or extra volume fused to the object's feet. We propose the opposite view. Under a dominant light source such as the sun, a cast shadow is a silhouette of the moving object seen from the light's direction. In effect it is an extra camera that the capture rig did not pay for. In the sparse-view regime, where geometry is badly under-determined, that extra viewpoint is exactly the missing constraint.
Concretely, we will build a dynamic 3D Gaussian Splatting model. 3D Gaussian Splatting represents a scene as millions of small colored, semi-transparent 3D ellipsoids that are projected ("splatted") onto the image and blended, which renders at hundreds of frames per second. In our model, the static background's geometry is held fixed. Its brightness is darkened by an analytic shadow term computed from the moving Gaussians themselves. Gradients from the observed shadows flow back into the shape of the moving object. Computing the shadow costs one extra splatting pass from the light's viewpoint, so the model stays real-time. It also needs no human body template, unlike the closest prior method.
Core Claim
Our central claim is that explicitly rendering moving cast shadows as light-view transmittance of the dynamic Gaussians does two things in sparse static-camera capture. First, it removes shadow-induced shape distortion: no ghost volumes on the ground, and foreground/shadow separation that matches ground truth. Second, it measurably reduces sparse-view geometric artifacts on the moving object itself, because the shadow supervises the object's shape from an otherwise unobserved direction. Both should hold at interactive frame rates. The second effect is what is new. Prior shadow-aware neural rendering uses shadows to improve relighting and appearance, while prior sparse-view methods regularize geometry with smoothness, depth priors, or stereo matching. None of them uses the shadow as geometric supervision.
Our input directions play different roles. Shadow-induced distortion is the phenomenon we model. Sparse-view artifacts are the regime where modeling it pays off. The static-camera assumption is the operating constraint we accept: fixed sparse cameras, as in surveillance, sports, and lab rigs. We also relax its companion assumption that the background's appearance is constant. Rendering latency is the budget the method must respect.
Why Prior Work Leaves This Open
The closest work is Gaussian Shadow Casting for Neural Characters. It fits anisotropic Gaussians to a human skeleton and evaluates shadow-ray density in closed form, adding about 0.3 s per frame instead of the doubled cost of volumetric shadow ray marching. It reaches 22.04 dB novel-pose PSNR on synthetic sunlit animations, versus 17.52 dB for DANBO. Its shadow proxy is tied to a parametric body, however, and sits on a slow NeRF backbone that takes seconds per frame. It also uses the shadow to fix appearance (keeping shadows out of the albedo) rather than to constrain geometry under sparse views. We reuse its key technical insight, that Gaussians make shadow transmittance analytic. We drop the skeleton by deriving the proxy directly from the dynamic Gaussians, and we turn the shadow into a geometry signal.
Two other works constrain our design. SSD-GS decomposes 3D Gaussian Splatting into diffuse, specular, scattering, and shadow terms by tracing secondary rays through the Gaussian field. Its setting is static objects under dense one-light-at-a-time capture, and the ray tracing costs real-time performance. That tells us per-pixel secondary ray tracing is the wrong primitive for our latency budget. P2GS models per-view exposure and tone curves in linear radiance for driving scenes. It treats the scene's radiance as constant, so it explains global brightness changes but not local moving shadows. Our shadow term is the local complement to its global photometric model.
On the dynamic side, SWinGS learns per-Gaussian blending between rigid static Gaussians and deformable dynamic ones within flow-sized sliding windows. It reaches 31.10 dB PSNR at about 71 FPS on N3DV, but only with 21 synchronized cameras and no shadow mechanism. A moving shadow on the counter must therefore be absorbed either by deforming "static" Gaussians or by baking it into their color.
The sparse-view literature supplies our regularization baselines. RegNeRF penalizes depth roughness from unseen poses (18.89 dB PSNR with 3 views on DTU). Depth-aware few-shot Gaussian Splatting anchors Gaussians with monocular depth. EFA-GS keeps Gaussians from shrinking into high-frequency floaters early in training. The file notes that EFA-GS treats dark shadows as noise and can misfire on flat shadowed ground. All of these methods address the ambiguity between shape and appearance with priors. None of them adds a new observation.
The motivation for the shape-distortion side comes from three sources. Visual-hull systems project floor shadows into "ghost" volumes (Guðmundsson et al.). Unsupervised 3D object decomposition absorbs shadows into object slots and loses segmentation accuracy as a result (ObPose). Motion-only decompositions cannot separate an object from its shadow at all, because both follow the same trajectory (Huang et al., Hofmann et al.). Our model breaks that tie with physics instead of motion. The shadow must be consistent with the occluder's geometry and the light direction.
Research Program
The scene is split into static Gaussians, with fixed position and shape over the whole sequence, and dynamic Gaussians, which deform over time. This uses the per-Gaussian static/dynamic weighting of SWinGS. Static Gaussians carry an albedo and a normal. They do not carry a time-varying color. Their rendered color at time $t$ is
$$c_t(x) = a(x)\,\big(L_{\mathrm{amb}} + L_{\mathrm{dir}}\,\max(0, n(x)\cdot \ell)\,(1 - O_t(x))\big),$$
where $a(x)$ is the albedo of a static point $x$, $n(x)$ its normal, $\ell$ the unit direction toward the dominant light, $L_{\mathrm{amb}}$ and $L_{\mathrm{dir}}$ ambient and direct intensities, and $O_t(x)$ the opacity accumulated by dynamic Gaussians along the ray from $x$ toward the light at time $t$.
The key engineering step is computing $O_t$ by splatting, not ray tracing. Each frame, we cluster the dynamic Gaussians into a few hundred coarse proxy Gaussians by moment matching. We then splat them orthographically onto a plane perpendicular to $\ell$, using the same rasterizer as the primary render, and look up $O_t(x)$ by projecting each static point onto that plane. This is a soft shadow map. It is differentiable with respect to the dynamic Gaussians' means, covariances, and opacities, and it costs one small extra rasterization pass. No skeleton is involved: the proxy is whatever shape the dynamic Gaussians currently have.
The light parameters $\ell$, $L_{\mathrm{amb}}$, and $L_{\mathrm{dir}}$ are optimized jointly with the scene. We initialize them from the dark-region statistics of the static camera feeds, since fixed cameras make a per-pixel temporal median background cheap to compute. Two known failure modes of such decompositions need countermeasures. The first is shadows leaking into albedo. Following the "grey loss" of Gaussian Shadow Casting and the staged schedule of SSD-GS, we fit geometry and lighting before albedo is allowed to vary freely. The second is the opposite leak, where the model removes real dark texture as "shadow". We address it by requiring $O_t$ to come only from dynamic Gaussians, so a static dark patch has no occluder to blame.
The research questions are threefold. The first is whether the shadow gradient into dynamic geometry actually reduces sparse-view artifacts on the object, not just on the floor. The second is how this benefit scales with shadow contrast and with the angle between the light direction and the real cameras. We predict the largest gains when the light is far from every camera, because the shadow is then most informative, and near-zero gains under diffuse indoor light. The third is how coarse the proxy can be before shadow edges, and the geometry they supervise, blur. Gaussian Shadow Casting reports exactly this blur for fingers and clothing folds.
Baselines
The naive reference is a dynamic 3D Gaussian Splatting model trained on the same two to four cameras, with no shadow model and no extra regularization. It shows how badly the problem manifests: ghost volumes, floor floaters, and object shape errors.
The strongest general dynamic competitor is SWinGS. With all 21 N3DV cameras it reports 31.10 dB PSNR at about 71 FPS. We found no reported result on Lacuna for SWinGS or any dynamic Gaussian method in a two-to-four-camera setting, so we will rerun it ourselves on the camera subsets we use, with its released hyperparameters and the same per-scene iteration budget as our method. Beating it means lower geometric error on the dynamic object and on the floor, and lower foreground/shadow segmentation error, with novel-view PSNR at least matching. A PSNR gain alone would not count, because a model can fit images well with wrong geometry.
The strongest shadow-aware competitor is Gaussian Shadow Casting, which reports 22.04 dB novel-pose PSNR on its synthetic sunlit benchmark. It can only run where a body skeleton exists, so we compare on its own human sequences under its protocol. There, beating it means matching its PSNR while rendering at interactive rates rather than seconds per frame. On non-human sequences (vehicles, animals, carried objects) it cannot run, and that inapplicability is itself the comparison.
We also run a strengthened sparse-view baseline: SWinGS plus monocular-depth anchoring in the style of depth-aware few-shot Gaussian Splatting, plus EFA-GS-style expansion against early shrinkage. This represents "fix floaters with priors" at full strength.
The decisive ablation keeps our shadow rendering but stops gradients from the shadow map into the dynamic Gaussians. Shadows still render and the floor is still explained, but the shadow no longer supervises object shape. If this ablation matches the full model on object geometry, our central claim fails, and the method is merely a better appearance model. A second ablation fixes the light direction to ground truth, to separate light-estimation error from modeling error. All methods share data, initialization, iteration count, and GPU, and every baseline gets the same hyperparameter search budget as our method.
Why not just…?
Why not just mask shadows with a 2D shadow detector and drop those pixels from the loss? This is the most serious alternative, and it is cheap: FSDNet runs at 77.52 FPS (Revisiting Shadow Detection). It has two weaknesses. First, detectors are unreliable exactly where shadows are hard: on that paper's CUHK-Shadow benchmark, DSDNet's balanced error rate rises from 2.17% on ISTD to 8.09%, and BDRAR's from 2.69% to 9.13%. Second, masking discards the shadow's information instead of using it, so it cannot reduce object geometry error. We will run this baseline, and also a variant that uses the masks as extra supervision for our shadow map. If masking matches us on object geometry in the stop-gradient comparison, we will report that the floor cleanup is the only real gain.
Why not just give each static Gaussian a time-varying color or per-frame appearance code? That is effectively what SWinGS's flexibility and P2GS-style brightness modules allow. It will fit the training images, and possibly score well on held-out cameras at the same timestamps. But it ties shadow appearance to particular frames rather than to the occluder, so it cannot predict shadows for novel poses or new light directions, and it adds no geometry constraint. Our experiments test exactly these cases: held-out time segments with new configurations of the moving object, and object geometry error.
Why not just add stronger depth or smoothness priors? Priors such as RegNeRF's unseen-view depth smoothness or monocular depth do reduce floaters, and they may already remove much of the floor ghosting. They cannot recover the back side of a moving object that no camera sees, whereas a low sun can. They are also complementary to our method rather than exclusive. We therefore report our method both alone and stacked on the strengthened prior baseline. If the stacked gain over the prior-only model is negligible, priors suffice and the shadow-as-camera idea does not earn its complexity.
Proposed Experiments
The primary benchmark must have ground-truth dynamic geometry and a controllable sun, and none of the datasets in the file has both. We will therefore render synthetic sequences in a simulator such as CARLA or a standard renderer: walking people, vehicles, and animals, captured by two to four static cameras, with sun elevation and azimuth swept so that the light-to-camera angle varies systematically. We measure Chamfer distance of the dynamic object against ground truth, ghost volume on the ground plane, shadow balanced error rate of the rendered shadow map, and foreground segmentation with shadows labeled as background, following the ObPose and CDnet convention. We also report PSNR, SSIM, and LPIPS on held-out cameras, and frames per second.
The decisive check is a plot of object geometry improvement against the light's angular distance from the nearest camera. Our hypothesis predicts a monotone increase, with no improvement under diffuse light. Seeing that curve, and seeing it disappear under the stop-gradient ablation, would establish that shadows act as a camera.
On real data we will use three sources. The first is the Gaussian Shadow Casting outdoor sunlit sequences, for the head-to-head comparison against its 22.04 dB. The second is N3DV subsampled to three and four cameras, a soft-shadow indoor stress test where we expect small gains, which we will report honestly. The third is the CDnet 2014 Hard Shadow category (CDnet 2014), used as a 2D check that our rendered foreground masks separate objects from shadows. That benchmark's PTZ category, where median F-measure is 0.14, shows that moving cameras remain out of scope. Extending the method to moving cameras, with poses estimated jointly as in the bilayer segmentation of Jiang et al., is a natural follow-up but not part of this proposal.
For latency, we will measure the overhead of the shadow-map pass against proxy count and resolution. The target is to keep the full renderer within interactive rates on the same GPU as SWinGS, but we commit only to reporting the measured trade-off curve.
References
- Gaussian Shadow Casting for Neural Characters
- SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting
- P2GS: Physical Prior-guided Gaussian Splatting for Photometrically Consistent Urban Reconstruction
- SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting
- RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs
- Low-Frequency First: Eliminating Floating Artifacts in 3D Gaussian Splatting
- TOF imaging in Smart room environments towards improved people tracking
- ObPose: Leveraging Pose for Object-Centric Scene Inference and Generation in 3D
- Background Subtraction Using Low Rank and Group Sparsity Constraints
- Dense spatio-temporal motion segmentation for tracking multiple self-occluding people
- Revisiting Shadow Detection: A New Benchmark Dataset for Complex World
- CDnet 2014: An Expanded Change Detection Benchmark Dataset
- 3D Reconstruction of Dynamic Scenes with Multiple Handheld Cameras