Letting Category Templates Correct Their Own Masks in Dynamic Scenes

Most systems that reconstruct moving people and objects from video depend on a 2D segmentation model, a network that labels each pixel as "person," "chair," or "background," and they treat its output as ground truth. When the mask is wrong, the error goes straight into the 3D shape. A missed chair leg disappears. A mask that bleeds into the hand gripping the chair fuses hand and chair into one blob. Background pixels that leak into the mask become floating debris. This proposal reverses the dependency. Each moving object is represented as a deformable category template: a learned average shape for its class, such as "chair," warped by a small per-instance shape code. That shape code is held fixed across the whole video, and only the object's pose changes from frame to frame. Because one persistent, complete shape has to explain every frame, the template's rendered outline can overrule a bad mask in any single frame. The masks become quantities the system estimates jointly with the geometry, not inputs it must obey.

The promise is to recover complete, category-consistent geometry of moving objects from casual RGB video, including parts that are occluded or where the upstream segmenter fails. Current methods cannot do this. Methods that isolate moving things treat them as noise and throw them away. Methods that model the motion have no notion of what a chair should look like. The one method that uses templates for interacting objects requires an exact pre-scanned model of each specific instance.

Core Claim

A category-level deformable template with a time-invariant shape code supplies enough shape evidence to re-estimate unreliable 2D masks during dynamic reconstruction. Doing so should make object geometry degrade much more slowly as segmentation quality worsens than it does in either template-free dynamic reconstruction or rigid exact-template tracking. The new element is not templates in dynamic scenes as such, since BEHAVE already tracks objects with templates. It is two things together: the template is category-level, and the segmentation is a latent variable that the template is allowed to correct. The measurable signature is the slope of reconstruction error against controlled mask corruption.

Relation to Prior Work

The motivating evidence comes from the segmentation literature. Mask errors are known to produce specific geometric failures: foreground objects flattened onto walls when segmentation misses them (Yang et al.), ghost geometry from leaking masks in avatars (Zhang et al.; Shao et al.), and over-carved volumes when silhouettes disagree across views (Yi et al.). All of these treat the mask as fixed. We treat it as something to be revised.

The closest dynamic systems split into two camps.

Template-free dynamic reconstruction. TFS-NeRF reconstructs interacting entities from sparse or monocular video. It uses mask-guided ray sampling, an invertible network that maps points back to a canonical space, and a separate signed-distance network per entity. It is deliberately template-free and so produces noisy surfaces under heavy occlusion. Split4D decomposes 4D scenes by clustering Gaussians with identity features supervised by per-frame masks. It smears boundaries at contacts and has no shape prior to fill in missing parts. Deformable 3D Gaussians and Dynamic 3D Gaussians deform scene-specific particle clouds with no category structure. DynaGSLAM separates dynamic Gaussians using optical-flow masks, and errors in those masks spawn floating Gaussians.

Template-based systems. BEHAVE tracks humans with the SMPL body model and objects with exact pre-scanned meshes, using multi-view RGB-D input. Chord deforms category implicit templates for hand-held objects, but works from single images with no time axis.

We take the deformation machinery from the deep-implicit-template line. That means a canonical signed-distance function (a network that outputs the distance to the surface at each 3D point) warped by a latent-conditioned field, as in DDIT and Chord. DDIT is a cautionary precedent: it depends entirely on fixed Mask3D segmentations and fails when they fail, which is exactly the dependency we remove.

Two further papers act as constraints on the design. ObjectMeshDeform shows that parameterizing deformation through Jacobians (the local stretch and rotation of each mesh face) suppresses the fold-overs that free coordinate offsets produce, and we adopt its regularizer. The known topological rigidity of templates, where a chair may or may not have armrests, bounds which categories we can claim.

Research Program

Each entity $k$ in a video has three parts. The first is a category template $T_{c_k}$, pretrained on category shape collections. The second is a single shape code $z_k$ shared over all frames. The third is a per-frame pose $\theta_{k,t}$: rigid for objects, articulated SMPL parameters for people. The background is a static Gaussian-splat scene, built by rendering many small 3D Gaussian blobs, with delayed densification as in RobustSplat so that moving entities do not leave permanent floaters behind.

For every frame and every pixel $p$, we keep a soft ownership variable $m_{k,t}(p)$, meaning the probability that the pixel belongs to entity $k$. These are optimized alternately with the geometry, in the spirit of expectation-maximization. The objective is

$$ \mathcal{L}=\sum_{t,p}\Big[\sum_k m_{k,t}(p)\,\rho\big(I_t(p)-\hat I_{k,t}(p)\big)\Big]+\lambda\sum_{k,t}\mathrm{BCE}\big(m_{k,t},S_{k,t}(z_k,\theta_{k,t})\big)+\mu\sum_{k,t}w_{k,t}\,\mathrm{BCE}\big(m_{k,t},M_{k,t}\big)+\mathcal{R}_J(z_k). $$

The first term is the photometric fit. $I_t$ is the observed frame, $\hat I_{k,t}$ is entity $k$'s rendering, and $\rho$ is a robust penalty. The second term asks the ownership map to agree with $S_{k,t}$, the silhouette of the deformed template rendered at the current pose. The third term asks it to agree with $M_{k,t}$, the off-the-shelf mask from a model such as SAM. That agreement is weighted by $w_{k,t}$, a per-frame reliability learned from how well the template and the photometric term agree with that mask. The last term, $\mathcal{R}_J$, is the Jacobian regularizer on the warp.

Because $z_k$ is shared across time, a frame where the mask cuts off a chair leg is outvoted by the frames where the leg is visible. At contact regions, the human template and the object template compete for pixels, so the boundary is settled by whichever shape explains those pixels better, not by the segmenter's guess.

The work has three parts. First, we build and pretrain the category templates. Second, we implement the joint optimization on monocular RGB. Third, we run a controlled mask-corruption study. We restrict claims to rigid categories that have shape collections, such as chairs, tables, boxes, and backpacks. Deformable items like yoga mats are reported separately as an expected failure.

Baselines

The naive reference point is our own pipeline with the ownership variables frozen to the SAM masks, i.e. $w=1$ and no template silhouette term. This is "templates, but trust the masks." It isolates the central ingredient, and beating it under corruption is the minimum result required.

The strongest template-free competitor is TFS-NeRF. It targets exactly this setting: interacting entities, sparse or monocular dynamic video, semantically separated geometry. On BEHAVE it reports a human Chamfer distance of 1.812 cm against roughly 3.9 cm for the next baseline. No object-specific number from TFS-NeRF was found on Lacuna. We will run it on identical BEHAVE sequences, with identical input masks and a matched GPU-hour budget, and report object Chamfer distance. Beating it means lower object Chamfer at every corruption level, with human error no worse than its reported 1.812 cm.

The strongest template-based competitor is BEHAVE itself, which reports an object vertex-to-vertex error of 21.20 cm. That figure uses multi-view RGB-D and exact instance scans, which is an advantaged setting. We will not claim to beat it directly. Instead we compare against a re-implementation that uses BEHAVE-style rigid tracking of the nearest retrieved category CAD model on our monocular input. This checks whether per-instance deformation adds anything over retrieval.

For mask-free decomposition, we include Split4D, whose reported decomposition mIoU on Multi-Human is 0.893. We will evaluate it with the same per-frame masks.

Two further ablations remove, in turn, the shared shape code (letting $z$ vary per frame) and the Jacobian regularizer.

Why not just…?

Why not just use a better video segmenter and feed its masks to TFS-NeRF? Temporally tracked masks remove flicker but cannot recover parts that are occluded in every frame. They also do not resolve contact regions, where the ambiguity lies in the image itself rather than in the tracker. This alternative is a real threat on lightly occluded sequences. The experiments settle it by giving every method the best available tracked masks as well as corrupted ones, and by reporting error separately on contact-heavy and occlusion-heavy frames.

Why not just add a robust loss that down-weights disagreeing mask pixels? Robust losses in the style of RobustNeRF, which reports 30.87 dB against 25.22 dB for Mip-NeRF 360 on its distractor benchmark, reject outliers but supply nothing to replace them with. A missing leg stays missing. We will add a trimmed mask loss to the template-free baseline as an explicit variant. If that variant matches us, the template contributes nothing and the proposal fails.

Why not just retrieve the closest category CAD model and track it rigidly? Retrieval gives completeness for free and may be enough for low-variation categories. The retrieval baseline above measures this directly. We expect deformation to matter mainly for high-variation categories such as chairs, and we will report results per category so that no average hides a null result.

Why not just generate a shape per frame with a generative prior, as in ShapeR, and fuse the results? ShapeR produces complete meshes from casual static captures. Per-frame generation, however, has no mechanism that forces the same instance to have the same shape across frames. We will test this by measuring shape variance across frames for a ShapeR-initialized variant.

Proposed Experiments

The primary benchmark is BEHAVE with monocular RGB taken from a single camera. Metrics are object and human Chamfer distance and vertex-to-vertex error against the pseudo-ground-truth fits, contact-region interpenetration, and mIoU of the re-estimated masks against annotated masks.

The decisive test is the corruption study. SAM masks are systematically eroded, dilated, dropped in a fraction of frames, and merged at human-object contacts, at five severity levels. For each method we plot object Chamfer distance against severity. The claim holds if our slope is clearly flatter than both TFS-NeRF and the frozen-mask ablation, and if the re-estimated masks score higher mIoU than the corrupted inputs.

A secondary check uses COMIC, with Chord's reported Chamfer distance of 7.69 as a reference. We will build short pseudo-sequences from its image collections to test whether temporal sharing of the shape code improves on single-frame deformation.

A negative result would also be informative. If the ablation with a per-frame shape code performs as well as the shared-code version, then persistence of shape across time is not what corrects the masks, and the idea should be abandoned in favor of plain robust losses.