CISyn: Human Mesh Recovery in Close Interactions with Physics-Simulated Synthetic Data

Yufan Deng, Lorenza Prospero, Christian RupprechtarXiv 2026 · arXiv

TL;DR

CISYN is a large-scale synthetic dataset designed to train AI models to recover 3D human meshes during close physical interactions like hugging or wrestling. By using a physics engine to simulate high-speed collisions between virtual humans, the researchers generated 300,000 frames of complex, entangled poses that are difficult to capture in the real world, leading to significant performance gains on interaction benchmarks.

What Problem Does This Paper Address?

3D Human Mesh Recovery (HMR)—the task of estimating a 3D body shape from a single 2D image—has become remarkably accurate for isolated individuals. However, it fails spectacularly when people interact at close range.

When two people hug, grapple, or dance, they create "pose entanglement": limbs overlap, bodies occlude one another, and depth ambiguities make it nearly impossible for a model to tell which arm belongs to whom. The primary bottleneck is data. Real-world "close interaction" datasets are tiny because they require expensive studio setups, while existing synthetic datasets usually place people independently in a scene, meaning they never actually touch or react to one another physically.

Methods

The authors bridge this gap using a two-pronged approach: a novel data generation pipeline and an uncertainty-aware training objective.

  1. Physics-Simulated Data (CISYN): Instead of recording actors, the authors use the Genesis physics engine. They "launch" two SMPL-X (a standard 3D human model) humanoids toward each other at various speeds and angles. The physics engine handles the collision dynamics, naturally producing contact, mutual occlusion, and physically plausible reactions. These sequences are then rendered in Blender with diverse clothing, lighting, and backgrounds.
  2. CISYN-HMR Model: The researchers built upon a state-of-the-art transformer-based model (MA-HMR). They added two key modifications to handle the messiness of close contact:
  3. Occlusion-Weighted Loss: Using ground-truth depth, the model down-weights the importance of joints that are hidden behind the other person during training.
  4. Uncertainty-Aware Laplacian NLL Loss: They added a dedicated "uncertainty head" that predicts a confidence score ($\sigma$) for every joint. If a joint is heavily occluded and the model is "unsure," the loss function allows it to express that doubt rather than forcing a potentially wrong guess that would introduce noise into the learning process.

Datasets

  • CISYN (New): 25,000 sequences (300,000 frames) of physics-simulated interactions with accurate 3D ground truth.
  • Training Mix: The model was trained on a combination of CISYN and "Base" datasets (AGORA, BEDLAM, and DTO-Humans).
  • Evaluation Benchmarks: The model was tested on real-world interaction datasets including Hi4D (social/casual), CHI3D (contact/combat), Harmony4D (sports/dance), and ExPI (extreme acrobatics).

Key Insights and Ideas

  • Physics as a Scalable Annotator: The "aha moment" here is that you don't need complex motion capture to get realistic interaction poses. Rigid-body collision dynamics naturally explore the "configuration space" of how two bodies can fit together, providing a diverse training signal that motion capture often misses.
  • Collision vs. Animation: Traditional synthetic data often looks like "ghosts" passing through each other. By enforcing physics, CISYN ensures that when a model sees a hand on a shoulder, the underlying 3D data reflects an actual physical touch.
  • Learning to be Uncertain: In close contact, some joints are mathematically ambiguous from a single viewpoint. The researchers found that letting the model "admit" it is uncertain (via the Laplacian NLL loss) actually helps it learn better from the joints it can see.

Key Results

  • Quantitative Improvements: Adding CISYN to the training mix yielded consistent improvements across all benchmarks. On the Hi4D dataset, the Mean Per-Joint Position Error (MPJPE) dropped from 61.2mm to 52.3mm.
  • Qualitative Superiority: Visually, the CISYN-trained model produces much tighter contact. Where baseline models might show an arm floating a few centimeters away from a partner's back, the CISYN model correctly "wraps" the limb around the torso.
  • Zero-Shot Generalization: Even though the physics simulation doesn't explicitly model "dancing" or "sports," the diversity of the collisions allowed the model to generalize to those specific activities better than models trained on real (but limited) data.

Limitations and Future Work

  • Passive Ragdolls: Currently, the virtual humans are "passive"—they don't have muscles or intent. They collide like ragdolls rather than hugging like people. The authors suggest adding active motor controllers (RL agents) in the future to simulate goal-directed interactions.
  • Interpenetration at Inference: While the training data is physically perfect, the model can still predict meshes that poke through each other at test time because there is no explicit "non-penetration" constraint in the neural network architecture itself.
  • Scaling Up: The current pipeline focuses on two-person interactions, though the authors demonstrated that it can be scaled to three or four people with minimal changes.

Summary updated

Abstract

3D human mesh recovery (HMR) in close-interaction scenarios is hindered by the lack of large-scale, accurately annotated training data: real capture is expensive, and existing synthetic datasets rarely include physical contact. We introduce CISyn, a large-scale synthetic dataset purpose-built for close human-human interactions, generated via physics simulation and Blender-based rendering. By simulating collision dynamics between SMPL-X-rigged humanoids, CISyn naturally produces the contact, mutual occlusion, and pose entanglement that characterize close-interaction scenes, with no manual annotation effort. The released set provides 25,000 sequences with accurate 3D annotations. We further propose an uncertainty-aware loss that allows models to learn more effectively from the heavily occluded examples in our data. Augmenting state-of-the-art HMR training with CISyn and replacing the regression loss with our uncertainty-aware variant yields consistent improvements on close-interaction benchmarks, validating CISyn as a scalable complement to real-world data. The dataset, generation pipeline, and code are available at https://github.com/yfde/CISyn.
BibTeX
@misc{deng2026cisyn,
  author = {Yufan Deng and Lorenza Prospero and Christian Rupprecht},
  title = {{CISyn: Human Mesh Recovery in Close Interactions with Physics-Simulated Synthetic Data}},
  year = {2026},
  eprint = {2611.18894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18894},
}
Figure