Motion-Induced Shading as the Signal That Separates Shape from Paint

A single photo cannot tell a raised bump on a surface from a painted dark spot. Both darken the same pixels. Models that split an object into geometry (its 3D shape) and texture (its surface color, or albedo) inherit this ambiguity. Fine surface patterns routinely leak into the shape as fake bumps, and real wrinkles get baked into the color map. Multi-view images of a static object do not fix this either. When the light is fixed relative to the object, the shading on a matte surface is the same from every camera angle, so a sculpted crease and a painted line still look alike.

Video of objects that move or deform does carry the missing information. When a head turns under a fixed lamp, or cloth folds, a real crease changes brightness because its surface direction changes relative to the light. A painted line keeps its color. We propose to use this asymmetry directly. We follow each surface point through the video with an off-the-shelf point tracker. We require its albedo to stay constant along the track, and we let all brightness change along the track be explained only by shading computed from the current geometry. The output is a feed-forward model that maps any video frame to a canonical shape, a per-frame deformation, a lighting estimate, and a canonical albedo map that can be edited. The core test is whether the model sends "sculpted" detail to geometry and "painted" detail to texture in cases where single-image and static multi-view methods cannot tell them apart.

Core Claim

Our claim is that temporal coherence supervision is useful for geometry-texture disentanglement in a specific way: as a source of variation in shading, and not mainly as a smoothness prior. We express each observed pixel color through a canonical coordinate $c$, the location in a shared object-centered space that a pixel tracks to across frames:

$$I_t(p) \approx a(c)\, s\big(n_t(c), L_t\big) + r_t(p),$$

The symbols are as follows. $I_t(p)$ is the color of pixel $p$ in frame $t$. $a(c)$ is a time-invariant albedo stored in the canonical space. $n_t(c)$ is the surface normal of the deformed geometry at frame $t$. $L_t$ is a low-order spherical-harmonic lighting estimate, meaning a compact smooth description of the incoming light. $s$ is the Lambertian shading function, the standard model of how a matte surface scatters light. $r_t$ is a small residual for specular highlights, kept small by a penalty.

Constancy of $a(c)$ along a track is the temporal-coherence constraint. The only path by which brightness can change over time is through $n_t$. As a result, any detail whose shading varies as the object moves relative to the light must be explained by geometry. Detail whose color is constant must be explained by albedo.

The predicted consequence is falsifiable. The benefit should grow with how much the object rotates relative to the light. It should vanish on clips where only the camera moves.

Relation to Prior Work

The closest work supplies parts of this pipeline but not the separation of shape from paint.

Densemarks trains a vision transformer on CoTracker3 point tracks from CelebV-HQ. The network maps every head pixel to a canonical 3D coordinate. It reaches a same-person matching error (MAE) of 3.68, against 7.60 for DINOv3. We reuse this recipe as our canonical-coordinate backbone. Densemarks itself stops at correspondence and learns no albedo, normals, or lighting.

Video Autoencoder shows that temporal continuity alone can separate static 3D structure from camera motion. It assumes a static world, however, and it keeps appearance and density fused in one voxel grid. Our signal requires the opposite: object motion relative to the light.

Vidu4D reconstructs deforming objects from one video using Gaussian surfels driven by a skinning field, reaching 27.30 PSNR. It optimizes each video for about an hour, and each surfel stores color together with geometry.

Flux4D separates moving content from static content rather than shape from texture, and it needs LiDAR.

Static disentanglement methods constrain the design.

  • CodeNeRF routes a shape code to the density layers of a neural radiance field and a texture code to its color layers.
  • Texture-GS maps 3D Gaussians to a UV texture space, a flat 2D image wrapped onto the surface. Its single-sphere unwrapping fails on shapes with holes.
  • TEGLO unwraps input pixels into a canonical texture space.

All three separate shape and texture through architecture alone, with no photometric signal that breaks the bump-versus-paint ambiguity. Texture-GS's seam problem is why we store albedo as a field over canonical 3D coordinates rather than on one UV chart.

Comprehensive Relighting motivates us from the other side. It splits human shading from appearance using normals and diffusion. Adding cycle-consistent temporal training cut its temporal L1 error from 6.552 to 4.019. Temporal supervision clearly matters for this decomposition. In that work, though, it serves only as a flicker penalty on 2D outputs, at about 25 seconds per frame.

Finally, the cycle-learning literature warns about degenerate solutions that close the cycle without carrying physical meaning:

Our analogous risk is concrete. A flat geometry with all detail in albedo, plus constant lighting, satisfies constancy perfectly on clips with little rotation. We address this by measuring it rather than assuming it away.

Research Program

We build a feed-forward network with a Densemarks-style transformer encoder, pretrained on CoTracker3 tracks from CelebV-HQ, that predicts three things per frame:

  • a canonical coordinate for every pixel;
  • a per-frame normal map, obtained by differentiating a deformation-plus-detail displacement field defined in canonical space;
  • spherical-harmonic lighting coefficients for that frame.

A small canonical albedo field is predicted per identity from a clip-level code.

Training combines four terms. The first is photometric reconstruction under the equation above. The second is a constancy term on $a(c)$ along each track. The third is the residual penalty. The fourth down-weights tracks that the tracker flags as occluded or uncertain, since tracker drift is a known failure mode of Densemarks-style supervision.

To guard against the flat-geometry collapse, we sample training clips stratified by estimated head rotation. We report results as a function of rotation, so that any collapse appears as a measurable regime and does not get averaged away.

We start with human heads because that is where tracks and canonical coordinates are known to work. Once the mechanism is confirmed, we extend to bodies on X-Humans.

Baselines

The naive reference point runs a static single-image decomposition on each frame independently and then smooths its albedo and normal outputs over time with optical-flow warping. The static method is a single-frame version of our own network, trained without the constancy term on the same data. This tells us whether any gain comes from temporal coupling or merely from filtering. It counts as beaten if our sculpted-versus-painted separation score on the synthetic diagnostic is clearly higher, not merely if flicker is lower.

The strongest cheap competitor is Densemarks plus texture baking. We take the trained Densemarks coordinates and set albedo at each canonical location to the median color along its tracks. This uses exactly our correspondence and data. It is the alternative most likely to look competitive, because track medians already remove much of the noise. It is run with the identical backbone, tracks, and training budget. We beat it if its baked albedo carries measurably more shading (sculpted detail misattributed to paint) than ours. The structural reason to expect this is that a median averages shading rather than explaining it. For Densemarks-style correspondence on the CelebV-HQ split, the reported number is MAE 3.68 (Densemarks). We require our coordinates to stay within noise of that, so that the decomposition does not come at the cost of correspondence.

The strongest existing reconstruction method is per-video optimization in the style of Vidu4D, which reports PSNR 27.30 on its single-video benchmark. Because its surfels fuse color and geometry, we give it the same Lambertian shading head and constancy loss, optimized per clip. This turns it into an upper bound on what our signal can achieve without amortization. Matching it within about 1 dB of reconstruction PSNR, at feed-forward speed and with comparable separation scores, counts as success. Exceeding it is not required.

The key ablation trains our model on camera-motion-only clips, where the object is static relative to the light. If separation does not degrade there, our claimed mechanism is wrong.

No reported albedo or shape-versus-paint separation result was found on Lacuna for any method. For those metrics, every number will come from our own runs.

Why not just…?

Why not just run a static disentangler per frame and smooth it over time? Smoothing removes flicker, but it cannot create information that no single frame contains. A per-frame model that cannot tell a bump from paint will be consistently wrong after smoothing. The material is also negative on post-hoc temporal fixes. The DAVIS benchmark paper (A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation) reports a gap between per-frame accuracy and temporal stability. Comprehensive Relighting needed learned temporal training to reduce temporal L1 from 6.552 to 4.019. Our naive baseline tests this directly on the separation score.

Why not just bake Densemarks correspondences into a texture by averaging colors along tracks? This is the real threat. On low-relief content under soft light it may do almost as well as our method. Its weakness is principled. The median of $a(c)\,s(n_t, L_t)$ over time equals albedo multiplied by the average shading, so creases and wrinkles end up in the texture and edits will carry them along. The synthetic diagnostic measures exactly this leakage, as a function of relief depth. If baking matches us there, we will report that the mechanism adds nothing beyond correspondence.

Why not just optimize each video separately with an intrinsic-decomposition loss? Per-video optimization can use the same motion-induced shading signal and may well match or beat us on quality. We include it as an upper bound rather than a competitor to defeat. The contribution in that comparison is amortization: Vidu4D-style fitting takes about an hour per video, while we aim for a single forward pass. This proposal does not claim that feed-forward beats optimization on fidelity.

Why not just use a foundation model? Decoupling Shape and Texture in SAM-2 shows that large segmenters carry a shape bias that has to be corrected with texture augmentation. Scaling 4D Representations shows that static and language-supervised models lag video-trained ones on geometric tasks, with its own model reaching EPE 0.24 on RealEstate10K pose. Neither model outputs albedo, normals, and lighting. A frozen 4DS encoder is therefore a reasonable backbone swap to try within our method, not an alternative to it.

Proposed Experiments

The decisive test is a controlled synthetic diagnostic built on ShapeNet-Cars and ShapeNet-Chairs, plus synthetic head meshes. We apply matched pairs of detail that are pixel-identical in one rendered frame: a displacement-mapped relief and a painted decal of the same pattern. We render the objects rotating and deforming under fixed environment lighting, and also with camera-only motion.

Ground-truth albedo and normals let us score separation in two ways. One is the correlation of predicted normal detail with the true relief. The other is shading leakage into predicted albedo. We plot both against relative object-light rotation. The mechanism predicts three outcomes:

  • near-zero separation at zero rotation for every method;
  • a rising curve for ours as rotation increases;
  • a flat curve for median baking.

On real data, we evaluate on CelebV-HQ. We check that correspondence MAE stays near the reported 3.68. We also run a texture-swap test: we replace the canonical albedo with a decal and re-render held-out videos. We measure temporal L1 in the style of Comprehensive Relighting, and we run a human preference study on whether wrinkles stay attached to the face rather than the decal. The paired-detail test above, rather than any single headline score, is what would establish the claim.

To check that the approach generalizes beyond heads, we extend to X-Humans and report geometry fidelity against FastAnimate's reported Chamfer distance of 0.609 cm (FastAnimate). We note that FastAnimate uses body-model priors and multi-view scans, which we do not.