Layer-Relative Coding for Streaming Clothed 4D Humans

When a moving person is captured as a single fused 3D surface, almost everything looks like it is changing. A shirt rides on a swinging arm, a jacket slides over the shirt, and skin deforms under both. In world coordinates every point of this fused shell moves on every frame, so a compressor sees little to reuse. Yet most of that motion is inherited. The shirt barely moves relative to the arm beneath it, and the skin under the shirt is invisible from every viewpoint. We propose to stream free-viewpoint video of clothed people as a stack of separate layers: body, skin, inner garment, outer garment. Each layer stores its motion relative to the layer it rests on. Each layer is then refreshed only when its own motion relative to that parent layer, or its visibility, actually changes.

The central bet is that much of the apparent dynamism in volumetric human video comes from fusing the layers together, and that separating them exposes temporal redundancy that current compressors cannot reach. This matters for telepresence and volumetric video. In those settings bandwidth is the binding constraint, and loose clothing is exactly where today's motion-decoupled codecs break down.

Core Claim

Current redundancy-exploiting 4D systems split the world into exactly two parts. 4D-MoDe sorts 3D Gaussians into static background and dynamic foreground. A 3D Gaussian here is one of the many small semi-transparent ellipsoids used by Gaussian splatting to render a scene. 4D-MoDe sends the background once and streams foreground motion through a single shared motion field. 4DGS-WAM similarly caches a static background bank and moves dynamic actors as rigid bodies plus residual corrections. In both systems the dynamic foreground stays one fused surface. 4D-MoDe's own limitations section says this foreground fails on fast, non-rigid motion such as loose clothing.

Our claim is that the right unit of temporal reuse is a physical layer, with motion expressed in the coordinate frame of the layer beneath it. A layered representation brings two sources of reuse that are hidden in a fused surface. The first is small residuals: a garment's motion relative to its parent is small and changes slowly. The second is visibility: inner surfaces covered by an outer layer need no updates while they are covered. When they are uncovered later, they already exist, because the layered model keeps them complete rather than leaving holes. The claim is falsifiable. If anchoring every Gaussian to a parametric body model and coding the leftover motion works as well as the full layer stack, then the extra garment layers are not doing the work, and we will say so.

Research Program

The representation builds on the layered-human line of work. That line runs from Multi-Garment Net, which first registered garments as separate meshes over a parametric body, to Disco4D, which anchors each clothing Gaussian to a triangle of the SMPL-X body mesh and adds a signed-distance penalty that keeps clothing from passing through the body. Disco4D supplies the anchoring and non-penetration machinery we reuse. It is also the clearest example of the gap: it runs its deformation network over every Gaussian at every timestep and never asks which ones changed. Disco4D also handles only one clothing layer. We generalize its anchoring into a tree. SMPL-X pose parameters drive the skin layer. The skin layer's surface gives local frames for the inner garment. The inner garment, or the skin where the inner garment is absent, gives frames for the outer garment. Each layer has a canonical set of Gaussians sent once as a keyframe, plus per-frame offsets expressed in its parent's local frames. We extend the penetration loss pairwise between adjacent layers, so the stack stays ordered.

Streaming then becomes a per-layer, per-frame decision. For layer $l$ at frame $t$, the encoder chooses an update $u$ among three options. It can skip, in which case the parent's motion carries the layer along unchanged. It can send a low-rank residual. Or it can send a full residual. The choice minimizes

$$ J_{l,t}(u) = D_{l,t}(u) + \lambda\, R_{l,t}(u), $$

where $D_{l,t}$ is the rendering error on that layer's visible pixels across training views, $R_{l,t}$ is the bits needed for update $u$, and $\lambda$ sets the rate–quality tradeoff. A layer whose Gaussians are fully covered by its children from all views has $D_{l,t}=0$ for every choice, so the encoder skips it automatically. This is the 3D counterpart of DyCoke's "recycle bin," which parks tokens rather than deleting them so they can come back when attention returns to them. Here covered skin is parked, not discarded, and reappears complete when a jacket swings open.

The low-rank option follows the observation in RSTR that different parts of a diffusion transformer tolerate skipped computation to very different degrees. RSTR uses a singular value decomposition (SVD) to cache features with rank chosen by sensitivity. We apply the same idea to residual motion, with rank chosen per layer.

The encoder decides causally, using only the current and past frames. This is a deliberate contrast with 4D-MoDe, whose lookahead window adds buffering latency that its authors flag as incompatible with live telepresence. We will report whether causality costs bitrate.

Two choices keep the work grounded and honest. First, we begin with oracle layer labels from 4D-DRESS, which provides vertex-level inner and outer garment annotations on real 4D scans. We then replace the oracle with labels from an automatic parser, to measure how much the gains depend on perfect segmentation. 4D-DRESS itself needed human correction on 3.2% of frames, so labeling errors are a live risk. Second, we treat the layer tree as fixed per sequence. Topology changes such as unbuttoning are the hard case flagged in the file's gap analysis. We will report them as a failure category rather than hide them.

Baselines

The naive reference point is a keyframe-only stream. Canonical Gaussians are skinned by SMPL-X pose alone, with no residuals sent. This gives the floor: the bitrate is essentially pose parameters, and the quality loss shows how much non-rigid clothing motion matters on each outfit type. At the other extreme, a per-frame independent Gaussian reconstruction gives the quality ceiling and the bitrate a naive system would pay.

The strongest existing competitor is 4D-MoDe. It is the most direct published attempt to exploit temporal redundancy in streamed Gaussian video by decoupling layers, though its layers are static and dynamic only. It also includes the strongest relevant tools: adaptive keyframe refresh, residual-triggered insertion of new Gaussians, and rate–distortion training. On N3DV it reports a storage cost as low as 11.4 KB per inter-frame and a 5.46× compression ratio over 4DGC. We found no reported result on Lacuna for 4D-MoDe, or any streaming codec, on 4D-DRESS. We will therefore run it ourselves on multi-view renderings of 4D-DRESS sequences, using the same camera rigs, Gaussian budget, entropy coder and training iterations as our method, and the same tuning budget over its rate parameter. We will also run both methods on N3DV, so our reimplementation can be checked against the published 11.4 KB figure before any human-specific comparison is trusted.

Two ablations isolate the key ingredient. The first, which we call body-anchored residuals, anchors every Gaussian to SMPL-X faces, as Disco4D does. It codes the leftover per-Gaussian offsets with the same three-way skip, low-rank or full decision, with Gaussians grouped into rate bins by motion magnitude instead of by garment layer. This is the most dangerous ablation, because it captures most of the "inherited motion" benefit without any garment segmentation. The second, world-space rate bins, clusters Gaussians by world-space motion statistics into as many rate classes as we have layers. It is a direct multi-rate generalization of 4D-MoDe. We also run our full system with occlusion skipping turned off, and with parent-relative coordinates replaced by world coordinates, to separate the two proposed sources of reuse.

Beating the baselines means the following. At matched held-out-view PSNR, our method must use clearly fewer kilobytes per frame than both 4D-MoDe and body-anchored residuals. The gap should appear on 4D-DRESS Outer outfits specifically, and it should survive replacing oracle labels with automatic ones. A gain only on Inner outfits, or only with oracle labels, would not count.

Why not just…?

Why not just use one monolithic 4D Gaussian field with a continuous deformation network? Such a model cannot skip covered inner surfaces, because it has none: covered skin either does not exist or is fused into the clothing. The file also gives evidence that fused dynamic surfaces degrade on real motion. In DreamMesh4D, unconstrained dynamic Gaussian baselines reach a reference-view PSNR of 31.06 on Consistent4D, against 37.04 for its surface-bound model, with roughly double the FVD. FVD (Fréchet video distance) is a measure of temporal inconsistency. 4D-MoDe's single shared motion field is the strongest form of this alternative, and it is run directly as a baseline.

Why not just anchor everything to the body model and code the residual? This is the real threat. For tight inner clothing it may capture nearly all the benefit, since a tight shirt is roughly a second skin. The layered model should win only where a garment's motion relative to the body is large but its motion relative to the layer beneath it is small or invisible. Loose outer jackets and dresses over a shirt are the cases we expect. The body-anchored residuals ablation tests this directly, split by Inner and Outer outfits. If the bitrate gap on Outer outfits is within noise, we will conclude that garment layers buy editability and amodal completeness but not compression, and we will report it that way.

Why not just cluster Gaussians by observed motion into several update rates? World-space motion clustering puts a tight shirt on a fast arm into the high-rate group, even though almost none of that motion is new information. It also offers no principled way to tell that a region is covered rather than static. The world-space rate bins baseline, and our own model run in world coordinates, show how much of the gain comes from the choice of reference frame. That is exactly the quantity this proposal claims matters.

Proposed Experiments and Decisive Checks

The main experiment builds rate–distortion curves on 4D-DRESS, reported separately for Inner and Outer outfits and for subjects held out from any tuning. We plot kilobytes per frame against PSNR, SSIM and LPIPS on held-out views. We add Chamfer distance against the ground-truth scans, both overall and per layer, and an interpenetration rate between layers. For scale, the dataset's reconstruction reference point is SiTH, which reaches a Chamfer distance of 2.110 on Inner and 2.322 on Outer outfits, as reported in 4D-DRESS. We use this only as an indicator of difficulty, not as a streaming baseline.

The decisive check is the Outer-minus-Inner difference in bitrate savings over body-anchored residuals. The hypothesis predicts a clearly larger saving on Outer outfits. A flat difference would refute the claim that the garment layers drive the gain.

A second check breaks the savings down by source. For each sequence we log what fraction of skipped layer-frames were skipped because of small parent-relative residuals, and what fraction because the layer was fully covered. We relate both fractions to how loose the clothing is.

A third check measures robustness and latency. We swap oracle labels for automatic parsing, and we compare our causal encoder against a lookahead version with 4D-MoDe's window size. This reports what live operation costs in bitrate.

Finally, a small editability test checks that the layering is real rather than cosmetic. We swap an outer garment between subjects while streaming and report the artifacts. The layered model should produce no holes where the jacket used to be, because the inner layer underneath is already complete. This connects to the uncovered-background holes that 4DGS-WAM patches ad hoc.

N3DV results act as a sanity anchor for the 4D-MoDe reimplementation. We do not expect layered coding to help in general scenes without a layer structure, and we will report that honestly if it does not.