Training Skinning Predictors on Simulated Scans, Testing Them on Real Ones

Animating a 3D character means deciding, for every point on or inside it, how strongly it follows each bone of a skeleton. These per-point "skinning weights" are usually painted by artists. Neural networks now predict them automatically, and recent work treats them as a continuous skinning field, meaning a function that returns weights for any point in 3D space rather than only for mesh vertices. These predictors are trained on clean, artist-made synthetic characters. They are increasingly used on real scans: photogrammetry, depth fusion, multi-view reconstruction. Real scans fuse an arm to the torso wherever they touched during capture. They have holes, and they leave floating fragments of hair, garments, and accessories. These are exactly the conditions under which current skinning priors are known to break.

We propose to treat this as a sim-to-real problem with a specific, testable cause. Our hypothesis is that most of the gap comes from the acquisition process, not from the shapes being unfamiliar. We test it in two steps. First, we simulate the scanning pipeline on synthetic rigged characters and transfer the exact ground-truth weights onto the degraded reconstructions through the continuous field. Second, we evaluate on real 4D captures using a measurement that needs no artist labels: we skin a real scan in one pose, move it to another captured pose, and compare it with the real scan of that pose.

Core Claim

A feed-forward continuous skinning predictor trained on synthetically scanned characters will generalize zero-shot to real scans much better than the same network trained on clean synthetic meshes. This is not true of networks trained on randomly corrupted meshes, so the structure of the corruption is what matters. The new pieces are two. The first is the training distribution: synthetic characters posed with deliberate self-contact, captured by virtual cameras, and fused, with labels defined in space rather than on vertices. The second is a real-data reposing benchmark that measures skinning quality against physically observed motion rather than against artist weights.

Why This Gap, and Why Now

Two lines of prior work motivate the proposal. SkinningNet predicts weights with a graph network trained on RigNetv1 and tests on production game assets. It reports that geodesic distances, which prevent leakage on watertight meshes, fail on non-watertight and disconnected geometry. A Geodesic Cut-Cell Prior for Neural Skinning builds a ray-cast volumetric graph that cuts nearly 30% of RigNet's deformation error. It also falls back to Euclidean distance on disconnected components, which reintroduces leakage. Its audit found that roughly a third of the Articulation-XL test set duplicated training characters. Both papers therefore study synthetic-to-"wilder synthetic" transfer with geometric priors. Neither trains on simulated acquisition artifacts, and neither measures error on real captured motion. We do both, and we use the cut-cell prior as a competitor and as an optional input feature.

The continuous-field formulation comes from LEAP and NiLBS, which predict weights at arbitrary points. We reuse that idea for a practical reason. Weights defined in space can be transferred from a clean synthetic rig onto any reconstruction of it, whatever its topology. Per-subject methods such as SCANimate, Invertible Neural Skinning, and VINECS fit a skinning field to each captured person's scans or video. Invertible Neural Skinning explicitly notes it cannot transfer zero-shot across bodies. We aim for a single forward pass on an unseen scan, and we use these per-subject methods as optimization-heavy upper references.

Research Program

Simulating the scanner. We pose synthetic rigged characters from the de-duplicated Articulation-XL split and from RigNetv1. Poses are sampled to include self-contact, such as hands on hips, arms against the torso, and crossed legs, because contact is what real fusion glues together. Each posed character is rendered by randomized virtual multi-view depth cameras with realistic depth noise and missing returns. The renders are then fused with a standard volumetric fusion and surface reconstruction step. This reproduces the artifacts that matter: merged contacts, holes under armpits and between fingers, thin structures that vanish, and floating fragments from accessories.

Transferring labels through space. Because the field is defined over 3D space, each point of the reconstruction gets its label from the nearest point of the original posed synthetic surface. That point's true weights are known. Labels stay correct inside a fused contact, since each fused point physically came from either the arm or the torso.

The predictor. The network takes a point cloud sampled from the scan plus the skeleton in its scan pose, and returns weights at arbitrary query points. It can therefore skin meshes, point clouds, or Gaussian splats alike. Training uses a weight-regression loss and a deformation loss under random bone motions.

Two analyses accompany the method. The first decomposes the remaining real-data error by artifact type: contact fusion, holes, and fragments. The second tests whether the gap shrinks as the acquisition simulation is made more faithful.

Baselines

The naive reference point is the same predictor architecture trained only on clean synthetic meshes, with identical data, iterations, and hyperparameter search. Any gain over it is attributable to the training distribution.

The strongest existing feed-forward competitor is RigNet augmented with the cut-cell prior from A Geodesic Cut-Cell Prior for Neural Skinning. It is the best published structural fix for leakage and for non-watertight inputs. The paper reports nearly 30% lower deformation error than plain RigNet on Articulation-XL, but as a relative figure. No reported result was found on Lacuna for any feed-forward skinning predictor's reposing error on real scans. We will therefore rerun it ourselves with its released settings, on the same real inputs. Beating it means a statistically reliable reduction in real reposing error, concentrated in contact and fragment regions, with no regression on the synthetic de-duplicated split.

For humans specifically, the practical competitor is to fit SMPL to the scan and copy each point's weights from the nearest body vertex. Many avatar pipelines do this. It costs only a body fit, so we run it on the same scans with the fits supplied with the data.

As an upper reference we report per-subject optimization in the style of SCANimate. It reports 0.570 cm scan-to-mesh distance on CAPE, versus 0.970 cm for CAPE's template model, but it trains on many scans of each person. That number is not directly comparable to our single-scan zero-shot setting; it bounds what more data buys.

The ablation that isolates the key ingredient keeps the same network and data volume but replaces simulated capture with generic geometric corruption: random holes, point jitter, and random fragment insertion. The corruption is calibrated to match the hole and fragment statistics of the simulated scans.

Why not just…?

Why not just use the cut-cell prior, which already targets messy geometry? It addresses non-watertight meshes, but its own paper reports that it falls back to Euclidean distance on disconnected components. It also cannot know that two touching surfaces fused by a scanner belong to different limbs, because after fusion they really are one connected volume. Running it on our real benchmark settles the question, as does feeding it as an input feature to our predictor. If it alone matches our numbers, the data contribution is unnecessary and we will say so.

Why not just copy weights from a fitted SMPL body? For minimally clothed humans this is a real threat and may win on body regions. Its known weakness is geometry far from the skin. SCANimate and CAPE both document the limits of a fixed body topology for clothing. It also does not exist for characters without a template. We will report error separately for body, loose-clothing, and accessory regions, and add a hand setting, so the comparison shows where the template suffices and where it does not.

Why not just randomize the geometry instead of simulating a scanner? This is the domain-randomization argument, and it is the ablation above. If calibrated random corruption closes the same fraction of the gap, our claim that acquisition structure matters is false. The likely difference is merged self-contact, which random holes and jitter never produce, so contact-region error is the measurement that decides it.

Proposed Experiments

Real reposing benchmark. We build it from CAPE raw scan sequences, which come with per-frame body fits that supply the bone transforms. For each subject we take a source frame, predict weights on the raw scan, and move the scan to the bones of a target frame. We then measure scan-to-scan Chamfer distance and the fraction of points that move with the wrong bone, with correspondence taken from CAPE's registrations. Source frames are stratified by whether they contain self-contact.

Multi-view captures. We repeat the protocol on DynaCap, where VINECS reports 3.034 cm Chamfer for its per-subject model against 3.306 cm for SNARF and 3.599 cm for RigNet. We also use MANUS-Grasps hand captures from MANUS, where fingers touching each other and the object test contact fusion at small scale.

Synthetic check. We evaluate on the de-duplicated Articulation-XL split to confirm that realism training does not cost clean-mesh accuracy.

Decisive checks. The proposal succeeds if three things hold. First, simulated-scan training cuts real reposing error clearly below both the clean-trained network and cut-cell RigNet. Second, it beats calibrated random corruption, especially in contact regions. Third, error falls monotonically as the virtual scanner is made more faithful. If generic corruption matches simulated capture, or if the gains vanish once the cut-cell prior is added, the acquisition hypothesis is wrong and the paper becomes a real-data benchmark with a negative result. That would still matter given the leakage found in current synthetic test sets.