When a 3D capture method reconstructs an object from photos, it has to decide whether each dark pixel is dark because the surface is dark (its albedo, the intrinsic surface color) or because light cannot reach it. These methods usually guess wrong in exactly one place: where the object touches something else. A mug on a table or a hand around a bottle sits in a thin band of near-total shadow. That band is dark, poorly observed, and geometrically sloppy. Reconstructions there often float a few millimetres above the support or sink into it. Inverse rendering then paints the contact shadow into the albedo, and the shadow stays frozen on the asset when it is moved or relit. This is the "baked-in lighting" problem.
We propose to use physical plausibility, meaning no interpenetration and no floating, as the missing boundary condition for separating material from lighting. First, a differentiable contact solver fixes the gap between the touching surfaces. Second, the occlusion inside that corrected gap is computed in closed form from 3D Gaussian proxies of the two bodies, rather than by ray marching. This does two things at once. It removes the main source of albedo contamination near contact, and it lets contact shadows update in real time when an object is re-posed, which cached-probe methods cannot do without recomputation.
Core Claim
In composed scenes, most of the error in albedo within the contact band comes from errors in the gap distance, not from errors in the lighting model. Correcting the gap with a non-interpenetration and contact constraint, then computing visibility analytically from that gap, should remove most of that error. It should also cost far less per frame than ray-traced visibility.
Write the observed radiance at a contact-band point as $I = a \, V(d) \, E$. Here $a$ is albedo, $E$ is the unoccluded irradiance from the environment, and $V(d) \in [0,1]$ is the fraction of that irradiance not blocked by the neighbouring body at gap distance $d$. Photometric losses alone see only the product $aV$, so the split between $a$ and $V$ is fixed by whatever $d$ the geometry happens to have. A 2 mm floating error makes $V$ too large, and the optimizer compensates with a darker $a$.
We constrain $d$ with a physical term:
$$ d \in [-\delta, \, 0] \ \text{on the contact set}, \qquad d \ge 0 \ \text{elsewhere}, $$
where $\delta$ is the allowed compliant penetration. We use $\delta = 0$ for rigid-on-rigid contact and $\delta \approx 2$ mm for skin, following ContactOpt, which showed that rigid zero-penetration models of human grasps produce floating hands. The contact set itself comes from a learned contact predictor.
There is a useful parallel with minimal geometric solvers. On the Instability of Relative Pose Estimation and RANSAC's Role showed that pose solvers break down near specific degenerate configurations, even on clean data. We treat the contact band as the analogous degenerate region of inverse rendering: the place where the albedo-versus-visibility split is worst conditioned. We intend to measure that conditioning directly, not merely assert it.
Research Program
The representation has two layers. A relightable 3D Gaussian splatting model handles appearance. Gaussian splatting represents a scene as millions of small colored ellipsoids that are projected onto the screen and blended, with no neural network run at render time. Each rigid body or hand additionally carries a sparse set of a few hundred volumetric Gaussians fit to its reconstructed interior. This second layer follows Real-time pose and shape reconstruction of two interacting hands with a single depth camera. That work showed the overlap integral between 3D Gaussians is closed-form and differentiable, fast enough to enforce non-interpenetration at 30 FPS. It used these Gaussians only as collision proxies with no appearance; we use the same proxies for both collision and occlusion.
Contact solving proceeds as follows. Given poses from any reconstruction or composition step, we minimize a penetration energy plus a contact-attraction energy over the relative pose, and for hands over the pose parameters.
- Penetration energy. The Gaussian overlap integral penalizes the bodies passing into each other.
- Contact-attraction energy. This term pulls predicted contact regions to zero gap. For hands, the target contact maps come from a DeepContact-style predictor trained on ContactPose.
- Smoothing. We borrow the soft-minimum distance idea from A Smooth Analytical Formulation of Collision Detection and Rigid Body Dynamics With Contact, which builds twice-differentiable contact fields, so the energy has no discontinuities.
Denser signed-distance penalties, as in Grasping Field, are more expensive and serve as an accuracy comparison. We do not treat the contact-versus-penetration trade-off as solved. We sweep $\delta$ and the attraction weight, and we report floating gaps and penetration depths alongside the appearance metrics.
Visibility is computed analytically. For each surface Gaussian inside the contact band, we evaluate transmittance along a fixed small set of hemisphere directions, weighted by the environment lighting. Each evaluation passes only through the $k$ nearest proxy Gaussians of the other body. Transmittance along a line through a 3D Gaussian has a closed form, given by an error function, so no ray marching is needed. Outside the band, the base relightable model's own shadowing is used unchanged.
Decomposition follows the staged schedule that SSD-GS and ExMesh++ found necessary to limit cross-talk between geometry, shadows, and materials. We fix the contact geometry first, then fit albedo with $V(d)$ held fixed in the band, then fit specular terms.
The conditioning analysis runs alongside. On synthetic scenes we compute the Jacobian of rendered pixels with respect to per-point albedo and visibility, and map its condition number as a function of $d$ and of how many views see each point. This tests whether the contact band really is the worst-conditioned region, and how much the constraint improves conditioning.
Baselines
The naive reference is the same relightable Gaussian model with no inter-object occlusion term. Albedo is fit freely, and contact darkening is expected to leak into it. This baseline sets the size of the problem. If its albedo error in the band is not clearly worse than its error elsewhere, the premise fails.
The strongest existing competitor for dynamic composition is ComGS. It caches irradiance and occlusion in probes placed near the surface and inpaints missing lighting with a diffusion model. It reports a 1.4 dB PSNR gain over prior methods on SynCom, 26 FPS rendering, and 36 s per edit. It is the right competitor because it targets exactly real-time composition with shadows. Its weaknesses for this question are that it needs re-caching when the object's pose changes and that it has no collision handling. We will run the released configuration on the same 16 SynCom compositions with the same reconstruction inputs and GPU. Beating it means two things: higher relighting PSNR and lower albedo error inside the contact band at equal or better FPS, and correct contact shadows after a pose edit without the 36 s re-cache.
For the static, accuracy-first setting, the strongest competitor is ExMesh++. It freezes a reconstructed mesh and traces secondary rays for one-bounce indirect light, and reports relighting PSNR 34.19 on Synthetic4Relight and Chamfer 0.58 on DTU. We run it on composed scenes by merging the meshes before its Stage II. It represents "physically correct visibility, but on uncorrected geometry."
SSD-GS is the ray-marched shadow competitor within Gaussian splatting. It reports 31.12 dB PSNR on the real-world Pixiu scene under one-light-at-a-time (OLAT) capture, where the scene is lit by a single point light at a time. It gives the latency comparison for visibility accuracy obtained by tracing rays through Gaussians.
For contact geometry alone, ContactOpt is the reference. It reports HO-3D mean per-joint position error (MPJPE) falling from 57.7 mm to 48.1 mm. We re-run it from identical initial poses to check that our Gaussian contact solver is at least as good geometrically.
Three ablations isolate the key ingredient. The first removes the contact constraint and computes analytic visibility on raw geometry. The second keeps the corrected geometry but replaces analytic visibility with ray-traced visibility, which separates the accuracy gain from the speed gain. The third compares $\delta = 0$ against the compliant 2 mm band for hands.
Why not just…?
Why not just ray-trace ambient occlusion on the reconstructed geometry, as ExMesh++ already does? This is the real threat. If reconstructed gaps are already accurate, the contact term adds nothing. Evidence that they are not accurate comes from Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting. There, meshes converted from Gaussians developed dents on reflective and textureless surfaces, and high-accuracy contact tracking succeeded only 17% of the time. That evidence is indirect. The first ablation settles it: ray-traced visibility on raw geometry versus on contact-corrected geometry, with everything else fixed. If the gap between them in contact-band albedo error is negligible, we will report that the physics term is unnecessary and keep only the latency contribution.
Why not just let a diffusion model inpaint or remove the shadows, as ComGS does for missing lighting? Learned shadow prediction has a documented weakness. SSD-GS attributes its improvement over neural shadow baselines (31.12 dB versus 30.11 dB on Pixiu) to physically grounded visibility, and ContactOpt's explicit optimization beat the purely learned RefineNet at millimetre-scale contact. Neither result tests delighting at contact specifically, so the direct comparison against ComGS on SynCom is the evidence that counts.
Why not just capture under many lights so the ambiguity disappears? Multi-light capture does break the albedo-visibility product in the lit regions. However, the contact band is shadowed under nearly every light direction, so extra lights add little information exactly there. OLAT rigs are also unavailable for in-the-wild and hand-object capture. We will test this on the synthetic OLAT scenes that SSD-GS uses by placing objects in contact and measuring band albedo error as the number of lights grows.
Proposed Experiments
The controlled study uses new Blender-rendered scenes. Stanford-ORB-style objects are rested on supports with ground-truth albedo, and we inject gap errors from −3 mm to +3 mm. This yields curves of contact-band albedo error against gap error for each method, plus the condition-number maps.
- SynCom. Metrics are relighting PSNR, SSIM, and LPIPS, frame rate, and latency after a pose edit.
- ContactPose, contact accuracy. Our solver's contact band is scored against the thermal ground truth using area under the curve. The paper reports 81.29% for PointNet++ with mesh features.
- ContactPose, albedo consistency. The dataset has 2,306 grasps of 25 objects with thermally recorded contact. We believe the objects were captured with plain surfaces so that heat residue would be visible, but we will verify this. If so, any albedo variation that correlates with the per-grasp contact map across grasps of the same object counts as leakage. This gives a real-data test that needs no ground-truth albedo.
- HO-3D v3. HO-3D_v3 reports annotation error falling to 8.12 mm. We use it to check that improvements in appearance do not come at the cost of joint error.
- Stanford-ORB. Stanford-ORB serves as a no-harm control on isolated objects. The reference there is NVDiffRecMC at PSNR-H 24.43.
The decisive checks are three. First, contact-band albedo error on synthetic data should drop by a large fraction, not a marginal one, relative to the uncorrected ray-traced baseline. Second, leakage correlated with contact on ContactPose should shrink. Third, contact shadows should update per frame under re-posing while rendering stays near ComGS's 26 FPS. Failure on the first check falsifies the core claim. Failure only on the third leaves an accuracy result without the latency result.
References
- ContactOpt: Optimizing Contact to Improve Grasps
- On the Instability of Relative Pose Estimation and RANSAC's Role
- Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
- ContactPose: A Dataset of Grasps with Object Contact and Hand Pose
- A Smooth Analytical Formulation of Collision Detection and Rigid Body Dynamics With Contact
- Grasping Field: Learning Implicit Representations for Human Grasps
- SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting
- ExMesh++: From Multi-View Images to Relightable UV-PBR Mesh Assets via Topology-Adaptive Reconstruction and Decomposition
- ComGS: Efficient 3D Object-Scene Composition via Surface Octahedral Probes
- Object and Contact Point Tracking in Demonstrations Using 3D Gaussian Splatting
- HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset
- Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark