in2IN: Leveraging individual Information to Generate Human INteractions

Pablo Ruiz Ponce, German Barquero, Cristina Palmero, Sergio Escalera, Jose Garcia-Rodriguez

TL;DR

The researchers introduce in2IN, a diffusion-based model that generates human-human interactions by conditioning on both a global interaction description and specific descriptions for each individual involved. By combining this with DualMDM, a novel blending technique that incorporates a single-person motion prior, they achieve state-of-the-art results with significantly better control and diversity in how individuals move during an interaction.

What Problem Does This Paper Address?

Generating realistic 3D human-human interactions from text is a "holy grail" for fields like animation, gaming, and robotics. However, it’s incredibly difficult for two main reasons:

  1. Inter-personal dynamics: Modeling how two people coordinate (e.g., a handshake or a boxing match) involves high-dimensional, interdependent movements.
  2. Intra-personal diversity: Current models often produce "generic" interactions. In reality, two people might salute each other in many ways—one might bow while the other waves. Existing datasets lack the detailed, individual-level annotations needed to train models for this kind of fine-grained control.

Methods

The authors propose a two-pronged approach to tackle these challenges:

1. The in2IN Architecture

Unlike previous models that only look at a single interaction prompt (e.g., "two people salute"), in2IN uses a Siamese Transformer architecture. It processes two noisy motion sequences simultaneously, sharing parameters to respect the commutative nature of interactions.

  • Dual Conditioning: It uses self-attention layers conditioned on individual descriptions (e.g., "person A bows") to model intra-person dynamics, and cross-attention layers conditioned on the overall interaction description (e.g., "they salute each other") to model the coordination between them.
  • Multi-Weight Classifier-Free Guidance (CFG): The authors developed a new sampling function that allows users to independently tune the "strength" of the interaction prompt versus the individual prompts during generation.
The in2IN architecture uses Siamese Transformers with specific attention modules for individual and interaction-level conditioning.

2. DualMDM Blending

To further boost diversity, the authors introduce DualMDM. This technique blends the output of the interaction-aware in2IN model with a "prior" model trained on a massive dataset of single-person motions (HumanML3D).

  • Weight Scheduling: Instead of a fixed blend, they use a time-varying weight scheduler. They found that individual dynamics are most important during the early stages of the diffusion denoising process, while interaction coherence becomes critical toward the end.
Figure 3. Different weights schedulers tested for DualMDM: Ex- ponential , Inverse Exponential, Constant, and Linear.

Datasets

  • InterHuman: The largest existing interaction dataset. The authors used GPT-3.5 to automatically "augment" this dataset, generating individual motion descriptions for every interaction in the set.
  • HumanML3D: A standard single-person motion dataset used to train the individual motion prior for DualMDM.

Key Insights and Ideas

  • LLMs as Annotators: Manually labeling individual motions in thousands of interaction clips is prohibitively expensive. Using an LLM to "deconstruct" a global interaction prompt into individual components proved to be a highly effective proxy.
  • The "Precision vs. Diversity" Trade-off: Adding individual descriptions actually reduces the mathematical "MultiModality" score because the model is now more constrained to follow specific instructions. However, this results in much higher Extrinsic Individual Diversity (EID)—a new metric the authors proposed to measure how well a model responds to varied individual prompts.
  • Timing Matters in Diffusion: The success of the exponential weight scheduler suggests that a model needs to "decide" what an individual is doing early on, and then "refine" how they interact with their partner as the motion becomes clearer.

Key Results

  • State-of-the-Art: in2IN outperformed previous models like InterGen and MoMat-MoGen on the InterHuman dataset across key metrics, including R-Precision (how well the motion matches the text) and FID (how realistic the motion is).
  • Qualitative Superiority: As seen in the figures below, in2IN produces motions that are much more physically plausible and semantically aligned than previous baselines.
Figure 5. Interaction Description: The two guys meet, grip each other’s hand, and nod in agreement. The X-axis represents time.

Limitations and Future Work

  • LLM Hallucinations: Because the individual descriptions were generated by an LLM, there's a risk that the text doesn't perfectly match the original ground-truth motion in the dataset.
  • Manual Tuning: The current DualMDM approach requires choosing a weight scheduler ($\lambda$). The authors suggest that future work could involve learning these blending strategies automatically.
  • Complexity: The multi-weight CFG requires quadruple sampling from the denoiser, which increases the computational cost during inference compared to standard methods.

Summary updated

Abstract

Generating human-human motion interactions conditioned on textual descriptions is a very useful application in many areas such as robotics, gaming, animation, and the metaverse. Alongside this utility also comes a great difficulty in modeling the highly dimensional inter-personal dynamics. In addition, properly capturing the intra-personal diversity of interactions has a lot of challenges. Current methods generate interactions with limited diversity of intra-person dynamics due to the limitations of the available datasets and conditioning strategies. For this, we introduce in2IN, a novel diffusion model for human-human motion generation which is conditioned not only on the textual description of the overall interaction but also on the individual descriptions of the actions performed by each person involved in the interaction. To train this model, we use a large language model to extend the InterHuman dataset with individual descriptions. As a result, in2IN achieves state-of-the-art performance in the InterHuman dataset. Furthermore, in order to increase the intra-personal diversity on the existing interaction datasets, we propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D. As a result, DualMDM generates motions with higher individual diversity and improves control over the intra-person dynamics while maintaining inter-personal coherence.
BibTeX
@inproceedings{ponce2024leveraging,
  author = {Pablo Ruiz Ponce and German Barquero and Cristina Palmero and Sergio Escalera and Jose Garcia-Rodriguez},
  title = {{in2IN: Leveraging individual Information to Generate Human INteractions}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
}
Figure