Filling In Hidden Cloth: Normal-Map Diffusion Priors Projected onto Thin-Shell Physics

When a camera films a waving flag or a moving skirt, parts of the fabric fold out of view, turn away from the light, or simply carry no texture to track. The best current monocular cloth trackers handle this by asking the reconstructed surface to obey thin-shell physics: real fabric resists stretching and bending, so implausible shapes cost energy. Physics alone, however, cannot say which fold the hidden fabric actually formed. A buckling sheet has many configurations with nearly equal energy. Separately, diffusion models trained on surface normal maps produce sharp, believable wrinkles. A diffusion model is a generator that learns to turn random noise into samples step by step. A normal map is an image whose three color channels store the direction each surface point faces. These models know nothing about mechanics and will invent creases that no fabric could hold.

This note proposes to let each side fix the other's weakness. A normal-map diffusion model proposes several completions of the hidden or textureless regions, conditioned on what is visible. The thin-shell energy then acts as a projection: each proposal is fitted by a physically valid surface, and the proposals that physics cannot support are discarded. In two sentences, the generative prior supplies the specific fold and the physics decides whether that fold can exist. The target is monocular tracking of deformable surfaces, where occlusion and missing texture are the main open failure modes.

Core Claim

The closest existing system is Thin-Shell-SfT. It tracks cloth from one RGB video using a known rest-state template mesh. The moving surface is a neural deformation field, meaning an MLP that maps 2D fabric coordinates to 3D positions. That field is regularized by a Kirchhoff-Love thin-shell energy and rendered through Gaussian splats attached to the surface. It reports an average Chamfer distance of 3.3 on the φ-SfT benchmark, against 3.93 for the previous best. The file itself notes that it has no geometric prior for textureless, shadowed, or self-occluded regions and uses no diffusion guidance.

On the generative side, Fantasia3D and HumanNorm show that rendering normals instead of RGB forces an optimizer to carve real geometry rather than paint shading onto flat surfaces. NeuSD shows that normal maps are a cleaner signal than RGB for completing missing surface regions. None of these three constrains its output by mechanics.

The claim here is narrow and testable. Adding a normal-diffusion completion prior to a physics-regularized neural cloth tracker should reduce error specifically in occluded and textureless regions. The gain should hold only when the prior is coupled to the shell energy by projection, meaning sample, fit under physics, then select. Simply adding a score-distillation term to the loss should not deliver it. Score distillation (SDS) uses a diffusion model's denoising direction as a gradient on a 3D shape.

What is new is not "physics plus diffusion" in general. Physics-informed diffusion models and physics-informed distillation already add PDE residuals to diffusion, but they work on regular grids of fluid or heat fields, not on curved shells seen through a camera. The new element is using shell mechanics as a filter on discrete generative hypotheses about hidden geometry, with normals computed analytically from the same chart the physics uses.

Research Program

The surface is a chart $\phi(u,v,\tau)$ from fabric coordinates $(u,v)$ and frame index $\tau$ to 3D, represented as a neural deformation field in the style of NeuralClothSim. Its normal is computed exactly by automatic differentiation, $\mathbf{n} = \phi_u \times \phi_v / \|\phi_u \times \phi_v\|$, which is how Physics-Informed Neural Networks for Shell Structures obtains its metric and curvature quantities. Projecting this field through the known camera gives a screen-space normal map, $N_\tau(\phi)$, at no extra modeling cost. The same derivatives therefore feed both the bending energy and the image the diffusion model sees. This is the bridge between rasterized normals and analytic shell geometry that the file identifies as missing.

The method has three stages per window of frames.

First, run the Thin-Shell-SfT objective to get a physics-consistent track. Compute a visibility-and-texture mask $M_\tau$ for each frame. A pixel is unreliable when it is occluded or when the local photometric gradient is too weak to support tracking.

Second, a normal-inpainting diffusion model draws $K$ completions $\hat N^{(k)}_\tau$ of the normal map in the masked region, conditioned on the rendered normals in the reliable region. The diffusion model starts from a 2D normal diffusion backbone in the spirit of HumanNorm's normal-adapted model and is fine-tuned to inpaint garment normals on held-out CLOTH3D sequences.

Third, each sample is fitted by solving

$$\phi^{(k)} = \arg\min_\phi \; \mathcal{L}_{\text{photo}}(\phi) + \alpha \sum_\tau \big\| M_\tau \odot (N_\tau(\phi) - \hat N^{(k)}_\tau) \big\|_1 + E_{\text{shell}}(\phi),$$

warm-started from the stage-one track. Here $\mathcal{L}_{\text{photo}}$ is the original rendering loss on reliable pixels, $\alpha$ weights the normal agreement, and $E_{\text{shell}}$ is the Kirchhoff-Love membrane-plus-bending energy.

The final track is the fit with the lowest total residual. Equivalently, it is the sample that physics can satisfy at least cost while still agreeing with the visible frames. Temporal coupling across the window matters. A region hidden in frame $\tau$ is usually visible in nearby frames, so a sample is rewarded only if the surface that realizes it connects to what was seen before and after.

The design follows from the tension the file describes between noisy, timestep-dependent SDS gradients and stiff shell energies. If SDS dominates, the shell buckles; if the physics dominates, wrinkles are smoothed away. Projection avoids weighting an unpredictable stochastic gradient against a stiff energy. Each target is a fixed image, and physics either fits it or reports a high residual, which is the signal we want.

Two analyses go beyond accuracy. First, we measure how often physics rejects diffusion samples, and what those samples look like: self-intersections, or creases sharper than the bending stiffness allows. This directly quantifies how "mechanically blind" normal priors are. Second, we test the speculation in the file that normal targets precondition ultra-thin shell optimization. We record epochs to convergence as thickness falls toward the $t/L = 10^{-4}$ regime where the shell-PINN paper reports needing up to 10,000 epochs. This is a secondary question, and a negative answer would not undermine the main claim.

Baselines

The naive reference point is Thin-Shell-SfT run unchanged, with the same template, frames, and iteration budget. It is the strongest published method on this exact problem and the system our method extends. Its reported φ-SfT Chamfer distance of 3.3 is the number to beat on the standard benchmark, and we will re-run it ourselves under our harder occlusion protocol. Beating it means a lower Chamfer distance on masked regions with no loss on visible regions, because a prior that degrades observed geometry is not worth having. We also give it the extra compute our method consumes, as additional iterations, so that our gain cannot be explained by more optimization.

The second baseline adds a feed-forward monocular normal estimator as a data term on all pixels, with physics kept. It uses the same normal loss and weight but replaces the inpainted samples with a single deterministic prediction. This is the cheapest way to inject normal information. We fine-tune it on the same CLOTH3D renders used for our inpainting model, so data access is matched.

The third baseline is direct SDS. The normal diffusion model's distillation gradient is added to the Thin-Shell-SfT loss, and its weight is tuned over the same grid we use for $\alpha$. It isolates whether projection, rather than the prior itself, is responsible for any gain.

Two ablations isolate the key ingredient. In the first, the shell energy in stages two and three is replaced by a generic Laplacian smoothness term of tuned weight; it tests whether mechanics specifically does the filtering. In the second, we set $K=1$, removing hypothesis selection. Our method counts as successful only if it beats both ablations on occluded-region error. That result would show that both physical selection and multiple hypotheses matter.

Why not just…?

Why not just use a monocular normal estimator instead of a diffusion model? This is the most serious threat, and on textureless but visible regions it may well match our method. A good estimator recovers orientation from shading, and the "Silk Paradox" in Learning Cloth Dynamics suggests that what hurts RGB reconstruction there is specular appearance rather than geometry. For fully occluded pixels, though, an estimator has no input to work from and outputs one blurred guess. A generative inpainter outputs distinct fold hypotheses that physics can choose between. We report error separately for visible-textureless and occluded pixels. If the estimator ties us on both, the diffusion component is unnecessary, and we will say so.

Why not just add SDS from a normal diffusion model to the tracking loss? It is simpler and closer to Fantasia3D. The concern is the documented conflict between stochastic distillation gradients and stiff shell energy, which tends to produce either smoothing or divergence. The direct-SDS baseline under a matched tuning budget settles this. If it matches projection, the simpler method wins.

Why not just simulate the hidden region with physics and skip the generative prior? That is what Thin-Shell-SfT already does implicitly. The shell energy picks some low-energy configuration, but buckling admits many. The occluded-region comparison against the unchanged baseline measures how much the prior adds. When fabric is hidden for long stretches, the honest outcome may be better plausibility, measured by self-intersection rate and physical residual, rather than better accuracy. We will report both.

Why not use a conventional FEM or mass-spring cloth simulator for the projection step? The projection must backpropagate through camera rendering and stay consistent with a continuous chart. The file also reports two drawbacks of discrete solvers: shell FEM suffers locking in bending (Physics-Informed Neural Networks for Shell Structures), and discrete wrinkle patterns depend on mesh resolution (NeuralClothSim). A differentiable discrete solver would still be a fair substitute, and we treat it as engineering rather than a claim.

Proposed Experiments

The primary benchmark is the φ-SfT dataset, evaluated with Chamfer distance in Thin-Shell-SfT's protocol. Its natural self-occlusions are too sparse to stress the method, so we add a controlled protocol. Synthetic occluders and texture removal are applied to masked regions for varying durations, which lets us plot error against how long a region stays hidden. The second testbed is CLOTH3D++ (Learning Cloth Dynamics), using held-out sequences disjoint from those used to fine-tune the inpainter. Its ground-truth garment meshes provide rest templates and dense ground truth, and its fabric labels let us report silk separately from cotton and denim. No reported result on Lacuna exists for template-based monocular tracking on CLOTH3D++, so all of its numbers will be produced by our re-runs.

Metrics are Chamfer distance and normal angular error, each split into visible-textured, visible-textureless, and occluded regions. We also report the self-intersection rate, the final shell energy relative to the ground truth's energy, and the fraction of diffusion samples rejected by physics.

The decisive check is the occluded-region error. Our method must beat unchanged Thin-Shell-SfT, the normal-estimator baseline, direct SDS, and both ablations, with visible-region error unchanged. If occluded error improves only when temporal context exists and vanishes for permanently hidden regions, that is still a useful and honest result. It would mean the prior bridges gaps in observation rather than inventing unseen geometry. If the Laplacian ablation matches full physics, the mechanics claim fails and the method reduces to a smoothed generative prior.