Programs that animate a virtual person inside a room learn from perfect room models. Examples are a system that walks a character to a chair and sits it down, or one that steps it over a box. Their training data pairs motion-capture recordings with hand-built digital replicas of the capture space. When such a generator is used in a real room, the room has to be reconstructed from video. Reconstructions of rooms where people and furniture moved during capture are wrong in structured ways. They contain phantom blobs of geometry left behind by passing people, holes carved where a person stood, and doubled ghosts of chairs that were pushed. A generator that has only seen clean geometry reads a phantom blob as a step, a hole as a drop, and a ghost as an obstacle. The results are characters whose feet float, sink, or slide across a floor that is not there.
We propose to close this gap by passing the generator's own training scenes through the same reconstruction pipeline it will face at deployment. We render the digital-twin rooms as video with the recorded people and moving objects inside them, reconstruct them with standard dynamic-scene methods, and train the generator to produce the original clean-geometry motion from the corrupted reconstruction. Two claims would be new. First, the synthetic-to-real gap for scene-conditioned motion generation lies largely in the structure of reconstruction artifacts rather than in visual appearance. Second, simulating those artifacts faithfully does better than both random geometric noise and after-the-fact geometry cleanup at reducing foot sliding and penetration. Performance is measured against the true scene.
Core Claim
Scene-aware generators condition on local scene geometry. In SceMoS this is a height map around the character plus a bird's-eye image. In TRUMANS it is a local occupancy grid, meaning a 3D voxel map marking which cells are solid. These methods report near-ceiling contact quality on clean data; SceMoS reports a Contact Score of 0.98 and a motion FID of 0.31 on the TRUMANS benchmark.
The failure modes of reconstruction from dynamic captures are well documented:
- Floaters. 3D Gaussian Splatting, a reconstruction method that represents a scene as many small colored blobs, adds blobs wherever the image error is high. Transient people therefore leave permanent mid-air floaters (RobustSplat).
- Holes. Carving humans out of depth maps leaves holes in the scene exactly where the person was (SHARE).
- Unplanned surfaces. Contact-driven reconstruction hallucinates support surfaces under the body (CRISP).
These artifacts are concentrated precisely where people walk and touch things. That is also where a generator's contact decisions are made. Our claim is that this spatial correlation is why generic robustness tricks underperform. We test it directly.
Research Program
Paired corrupted-and-clean data. TRUMANS provides 15 hours of motion capture in 100 scenes, with exact digital twins and part-level object motion (for example, drawers opening and chairs moving). For each sequence we:
- Render multi-view and monocular video of the twin with the captured body and moving objects present.
- Reconstruct the scene using four pipelines of increasing sophistication:
- naive static 3D Gaussian Splatting, which has no handling of moving content;
- transient-filtered splatting (RobustSplat);
- 4D decomposition into per-object moving parts (Split4D), keeping only the static component;
- monocular metric carving in the style of SHARE.
- Convert each reconstruction into the generator's native conditioning format, either height maps or occupancy grids.
Each training example then pairs corrupted conditioning with the ground-truth motion that was physically valid in the true room.
Training. We fine-tune a SceMoS-style tokenized planner and a TRUMANS-style autoregressive diffusion model on a mixture of clean and corrupted conditioning. The corrupted fraction and the mix of reconstruction pipelines are the main knobs.
Contact-reliability head. We add one lightweight component. The head predicts, for each cell of the local height map, whether it would support a foot. It is supervised by comparing the reconstruction to the true twin. Its output gates the height-map features the motion decoder sees. This is the only architectural change, and it gets its own ablation.
Evaluating contact on moving objects. TRUMANS scenes include moving objects, so contact must be judged relative to the thing being touched. A foot or hand in contact with a surface should have zero velocity relative to that surface, not zero world velocity. We define sliding as
$$ s_t = c_t \,\lVert \mathbf{v}_{\text{joint}}(t) - \mathbf{v}_{\text{surface}}(\mathbf{x}_{\text{joint}}(t), t) \rVert, $$
where $c_t \in \{0,1\}$ is the contact label at frame $t$, $\mathbf{v}_{\text{joint}}$ is the contacting joint's velocity, and $\mathbf{v}_{\text{surface}}(\mathbf{x}, t)$ is the ground-truth velocity of the scene surface at the joint's position $\mathbf{x}_{\text{joint}}(t)$. The ground-truth surface velocity is available from the twin's object motion. This generalizes the world-frame zero-velocity loss of Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control. Unlike that loss, it does not penalize a hand that correctly moves with a drawer.
The central evaluation rule. Every generated motion is conditioned on the corrupted reconstruction but scored against the clean geometry. This separates "the model trusted bad geometry" from "the model made a bad motion."
Baselines
The do-nothing reference is the published SceMoS and TRUMANS models, trained only on clean twins and run on each reconstruction. On clean data, SceMoS reports FID 0.31 and Contact Score 0.98 on TRUMANS. We found no result on Lacuna for either model on reconstructed or real-video geometry, so the size of the degradation is itself our first measurement.
The strongest geometry-cleanup competitor is CRISP's primitive fitting. It converts noisy point maps into planar cuboids. CRISP reports that this cut humanoid tracking failures in Isaac Gym from 55.2% to 6.9%. We apply it to the same reconstructions before feeding the clean-trained generator. The inputs, reconstructions, and evaluation are identical to our method; only the step between reconstruction and generator differs.
The strongest physics competitor is PhysDiff. It projects intermediate diffusion samples onto motions a simulated humanoid can execute, and it reports an 86% reduction in combined physical error on HumanML3D, measured on flat ground. We place the corrupted reconstruction in the simulator and apply PhysDiff-style projection to the clean-trained TRUMANS model. The projection uses the same late-step schedule PhysDiff found best, and we give it the same wall-clock budget as our method.
The generic-robustness baseline trains on clean twins with random height and occupancy noise. The noise is matched to our artifacts in total magnitude and coverage but placed independently of where the person is.
Ablations remove the reliability head, and restrict training to one reconstruction pipeline at a time. The single-pipeline runs test whether the gains transfer to reconstruction methods not seen in training.
What counts as winning. Beating a baseline means lower sliding $s_t$, lower penetration into the clean geometry, and a lower tracking-failure rate in Isaac Gym on held-out scenes reconstructed by a pipeline excluded from training, while keeping FID within noise of the clean-trained model on clean input. Tuning budgets are equalized throughout: the same number of fine-tuning steps and the same hyperparameter sweep size for every learned baseline.
Why not just…?
Why not just add random noise to the height maps? This is the most serious threat, because domain randomization worked for transferring LAGOON's quadruped policies from simulation to a real robot. Our argument is that reconstruction errors are correlated with the body's own trajectory. A phantom blob sits exactly on the path a person walked, so matched-magnitude independent noise teaches the wrong invariance. The matched-noise baseline settles this. If it reaches our sliding and penetration numbers on held-out pipelines, the structured-artifact claim is false, and we will report the cheaper recipe.
Why not just clean the reconstruction first? Primitive fitting in CRISP works when scenes are made of planes. CRISP itself assumes a static, rigid scene and fails on curved furniture. It also cannot tell a floater lying on a walkway from a real low obstacle, since both look like small boxes. Our per-scene breakdown will separate planar rooms, where we expect cleanup to be competitive, from scenes with curved or moved furniture.
Why not just add physics simulation? A simulator is only as good as the geometry it receives. If a floater is loaded as solid, PhysDiff's projection will faithfully make the character step onto it. Physics fixes impossible motions, not wrong beliefs about the scene. It also multiplies inference cost. Scoring against the clean geometry exposes this directly. We also test the combination of our training with physics projection, since the two may be complementary.
Why not just fine-tune on real scans with real motion? Real paired data with ground-truth motion in reconstructed dynamic scenes is scarce. TRUMANS needed a laboratory rig. Staged synthetic-then-real training as in WHAM is compatible with our approach and is not an alternative to it. Our contribution is the synthetic stage, which makes the corrupted-to-clean pairing possible at all.
Proposed Experiments
The controlled test holds out 20 TRUMANS scenes. We report sliding $s_t$, penetration and floating against the true twin, Contact Score, FID, and Isaac Gym tracking-failure rate following CRISP's protocol. Results are broken down by reconstruction pipeline, with one pipeline never seen in training.
The real-scan test follows the zero-shot protocol TRUMANS used on PROX, ScanNet, and ScanNet++. Here we generate from reconstructions of those scenes built from captured video, and score against the released scans.
The in-the-wild test reconstructs RICH sequences with a SHARE-style monocular pipeline and measures tracking failure and sliding in simulation.
Decisive checks. The claim fails if any of these hold:
- the clean-trained model degrades little on reconstructed geometry, in which case there is no gap to close;
- matched random noise equals structured artifacts on the held-out pipeline;
- the gains vanish on moved-furniture scenes, where the relative-velocity sliding metric and doubled ghost geometry are the specific stress test.
A positive result would show sliding and penetration errors that concentrate spatially on reconstruction artifacts for the baselines, and disappear there for our model.
References
- SceMoS: Scene-Aware 3D Human Motion Synthesis by Planning with Geometry-Grounded Tokens
- Scaling Up Dynamic Human-Scene Interaction Modeling
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- SHARE: Scene-Human Aligned Reconstruction
- Contact-guided Real2Sim from Monocular Video with Planar Scene Primitives
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
- PhysDiff: Physics-Guided Human Motion Diffusion Model
- LAGOON: Language-Guided Motion Control
- WHAM: Reconstructing World-Grounded Humans with Accurate 3D Motion