Learning 3D Human Interaction Under Data Scarcity

136 concepts116 papers1 proposal

In the quest to create embodied AI that can navigate a kitchen or avatars that can realistically wrestle, researchers face a fundamental wall: interaction data scarcity. While we have massive datasets for static images and single-person motions, capturing how humans physically engage with objects, environments, and each other in 3D is prohibitively difficult.

Traditional motion capture (mocap) requires expensive studio setups and markers, often resulting in "clean" but sterile data that lacks the messy complexity of the real world. As Joo et al. observe, most available resources are restricted to 2D views or single-person activities, failing to reflect the triadic or group dynamics essential for social AI. This scarcity forces a shift in strategy: if we cannot record the world in its entirety, we must learn to synthesize, lift, and compose interaction from the fragments we have.

The Dimensionality Gap

The core challenge is that interaction is not just about the human; it is about the spatial relationship between the human and an external entity. Liu et al. highlight that without large-scale datasets providing diverse, physically grounded poses within 3D environments, models struggle to learn the geometric affordances of a scene—like where a person can sit or stand.

This problem scales exponentially with complexity. For instance, Cho et al. note that training foundation models for hand-object interaction (HOI) is hindered by a lack of diversity in hand shapes and demographics. When you move to multi-person scenarios, the data becomes even thinner. Xu et al. point out that while two-person data exists, high-quality mocap for groups of three or more is almost non-existent, creating a bottleneck for simulating social scenes.

Lifting and Hallucinating Data

To bypass the mocap studio, researchers are turning to "lifting" 3D information from the vast ocean of 2D web videos. Zhang et al. introduced InterPose, which automatically extracts 3D motion sequences from tens of thousands of web videos. By leveraging the diversity of the internet, they enable models to generalize to complex real-world tasks that studio data simply doesn't cover.

Another approach is to "hallucinate" the missing data. Liu et al. proposed a framework to synthesize 1.5 million 3D poses within indoor scenes, training a model to predict where humans could be. Similarly, Cai et al. address the lack of data for moving objects in cluttered rooms by creating massive synthetic datasets that bridge the gap between static interaction and dynamic manipulation.

Compositional Synthesis

If we lack data for a specific complex interaction—say, two people carrying a heavy table—we can treat it as a composition of simpler, known interactions. Zhao et al. introduced COINS, which treats human-scene interaction as a composition of "atomic" action-object pairs. This allows a model to generate a person sitting on a sofa while touching a table, even if it never saw that exact combination during training.

This "synthesis-by-composition" logic extends to social dynamics. Wu et al. and Xu et al. demonstrate that by using two-person motion priors and graph-driven sampling, we can generate realistic multi-person animations without needing specialized group training data. The interaction is broken down into a graph of pairwise relationships, making the problem tractable despite the data shortage.

Injecting Physical and Geometric Priors

When data is sparse, the "laws of physics" act as a powerful regularizer. Models often produce "floaty" animations or "footskate" (feet sliding through floors) because they haven't seen enough examples of solid contact. Zhang et al. address this by enforcing ground contact constraints, while Li et al. use reinforcement learning to ensure characters obey physical laws like gravity and joint limits.

Geometry also provides a shortcut to generalization. Zhou et al. developed GEARS, which uses "joint-centered" local geometry sensors. Instead of trying to learn every possible object shape, the model learns how a hand relates to the local surface of an object. This allows it to interact with objects it has never seen before, effectively doing more with less data.

Scaling the Ground Truth

Finally, the community is building new, massive benchmarks to provide the "missing link" for future models.

  • HOGraspNet (Cho et al.) provides 1.5 million frames of diverse grasping taxonomies.
  • GigaHands (Fu et al.) offers 34 hours of bimanual hand activities.
  • WildO2 (Han et al.) moves beyond simple grasping to free-form interactions like poking and pushing in the wild.

By combining these massive new datasets with synthetic pipelines and compositional reasoning, researchers are beginning to overcome the scarcity bottleneck, moving toward AI that understands the physical nuances of the human world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interaction_data_scarcity_1c9c287f,
  title = {interaction data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interaction-data-scarcity/rdr_1c9c287f4cdb417f920af6ac6cc5ef38},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_interaction_data_scarcity_1c9c287f_v1,
  title = {interaction data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interaction-data-scarcity/rdr_1c9c287f4cdb417f920af6ac6cc5ef38/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake