TL;DR
The researchers introduce in2IN, a diffusion-based model that generates human-human interactions by conditioning on both a global interaction description and specific descriptions for each individual involved. By combining this with DualMDM, a novel blending technique that incorporates a single-person motion prior, they achieve state-of-the-art results with significantly better control and diversity in how individuals move during an interaction.
What Problem Does This Paper Address?
Generating realistic 3D human-human interactions from text is a "holy grail" for fields like animation, gaming, and robotics. However, it’s incredibly difficult for two main reasons:
- Inter-personal dynamics: Modeling how two people coordinate (e.g., a handshake or a boxing match) involves high-dimensional, interdependent movements.
- Intra-personal diversity: Current models often produce "generic" interactions. In reality, two people might salute each other in many ways—one might bow while the other waves. Existing datasets lack the detailed, individual-level annotations needed to train models for this kind of fine-grained control.
Methods
The authors propose a two-pronged approach to tackle these challenges:
1. The in2IN Architecture
Unlike previous models that only look at a single interaction prompt (e.g., "two people salute"), in2IN uses a Siamese Transformer architecture. It processes two noisy motion sequences simultaneously, sharing parameters to respect the commutative nature of interactions.
- Dual Conditioning: It uses self-attention layers conditioned on individual descriptions (e.g., "person A bows") to model intra-person dynamics, and cross-attention layers conditioned on the overall interaction description (e.g., "they salute each other") to model the coordination between them.
- Multi-Weight Classifier-Free Guidance (CFG): The authors developed a new sampling function that allows users to independently tune the "strength" of the interaction prompt versus the individual prompts during generation.
2. DualMDM Blending
To further boost diversity, the authors introduce DualMDM. This technique blends the output of the interaction-aware in2IN model with a "prior" model trained on a massive dataset of single-person motions (HumanML3D).
- Weight Scheduling: Instead of a fixed blend, they use a time-varying weight scheduler. They found that individual dynamics are most important during the early stages of the diffusion denoising process, while interaction coherence becomes critical toward the end.
Datasets
- InterHuman: The largest existing interaction dataset. The authors used GPT-3.5 to automatically "augment" this dataset, generating individual motion descriptions for every interaction in the set.
- HumanML3D: A standard single-person motion dataset used to train the individual motion prior for DualMDM.
Key Insights and Ideas
- LLMs as Annotators: Manually labeling individual motions in thousands of interaction clips is prohibitively expensive. Using an LLM to "deconstruct" a global interaction prompt into individual components proved to be a highly effective proxy.
- The "Precision vs. Diversity" Trade-off: Adding individual descriptions actually reduces the mathematical "MultiModality" score because the model is now more constrained to follow specific instructions. However, this results in much higher Extrinsic Individual Diversity (EID)—a new metric the authors proposed to measure how well a model responds to varied individual prompts.
- Timing Matters in Diffusion: The success of the exponential weight scheduler suggests that a model needs to "decide" what an individual is doing early on, and then "refine" how they interact with their partner as the motion becomes clearer.
Key Results
- State-of-the-Art: in2IN outperformed previous models like InterGen and MoMat-MoGen on the InterHuman dataset across key metrics, including R-Precision (how well the motion matches the text) and FID (how realistic the motion is).
- Qualitative Superiority: As seen in the figures below, in2IN produces motions that are much more physically plausible and semantically aligned than previous baselines.
Limitations and Future Work
- LLM Hallucinations: Because the individual descriptions were generated by an LLM, there's a risk that the text doesn't perfectly match the original ground-truth motion in the dataset.
- Manual Tuning: The current DualMDM approach requires choosing a weight scheduler ($\lambda$). The authors suggest that future work could involve learning these blending strategies automatically.
- Complexity: The multi-weight CFG requires quadruple sampling from the denoiser, which increases the computational cost during inference compared to standard methods.
Summary updated