TL;DR
This paper introduces MARRS, a novel framework for generating human reactions conditioned on another person's actions. MARRS addresses limitations of existing methods by using continuous representations, a unit-based approach for body and hands, and a novel mutual modulation mechanism to improve coordination.
What Problem Does This Paper Address?
Generating realistic human reactions to actions is a challenging task with applications in animation, gaming, and robotics. Current methods often rely on vector quantization (VQ), which can lead to information loss and codebook collapse. While autoregressive models and diffusion models have shown promise, they can be computationally expensive or struggle with fine-grained coordination between different body parts. Specifically, existing approaches often neglect the crucial aspect of mutual perception between distinct body units (like the torso and hands), leading to less coordinated and natural reactions.
Methods
MARRS tackles this by employing a two-stage approach:
- Unit-distinguished Motion Variational AutoEncoder (UD-VAE): This stage splits the whole-body motion into two distinct units: "body" and "hands." Each unit is then encoded independently using a VAE. This allows the model to learn representations for these parts separately, avoiding information loss associated with VQ.
- Masked Reaction Generation Model: This stage comprises several key components:
- Action-Conditioned Fusion (ACF): This module takes the active (actor's) motion and masked reactive (reactor's) tokens. It uses attention mechanisms to extract relevant information from the actor's motion and fuse it with the reactor's partially masked tokens.
- Mutual Unit Modulation (MUM): To address the lack of coordination between body and hands, MUM allows information from one unit to adaptively modulate the other. This bidirectional communication ensures that body and hand movements are synchronized and contextually appropriate.
- Diffusion MLPs: Instead of a single diffusion model for the entire motion, MARRS uses compact MLPs as noise predictors for each distinct unit (body and hands). This factorizes the problem, making optimization easier and improving learning efficiency. The diffusion loss is applied to model the probability distribution of each token individually.
Datasets
The paper evaluates MARRS on the following datasets:
- NTU120-AS: A large-scale dataset for human action-reaction synthesis.
- Chi3D-AS: Another dataset for human action-reaction synthesis.
- Inter-X: A dataset specifically designed for human-human interaction.
Key Insights and Ideas
- Unit Division is Crucial: Splitting the body into distinct units like "body" and "hands" significantly improves reaction synthesis, outperforming methods that treat the whole body as a single unit or use simpler divisions (e.g., upper/lower body).
- Mutual Modulation for Coordination: The MUM mechanism, allowing body and hands to influence each other, is vital for generating natural and coordinated reactions. Bidirectional modulation (body influencing hands and vice-versa) yields the best results.
- Continuous Representations Beat VQ: Avoiding vector quantization (VQ) and working with continuous representations directly addresses information loss and codebook collapse issues inherent in VQ-based methods.
- Factorized Diffusion for Efficiency: Using separate diffusion models for body and hands units, conditioned on actor motion, simplifies the generation process and improves learning.
Key Results
MARRS demonstrates state-of-the-art performance across various metrics, including FID (Frechet Inception Distance), Acc (Accuracy of action recognition), Div (Diversity), and Multimod (Multimodality).
- Quantitative: On NTU120-AS, MARRS achieved the best FID, Acc, and Multimod scores in both train-conditioned and test-conditioned settings. Similar superior performance was observed on Chi3D-AS and Inter-X datasets.
- Qualitative: Visualizations show that MARRS produces more plausible body movements, relative positions, and natural hand gestures compared to existing methods like ReGenNet. The generated reactions are more synchronized and realistic.
- User Study: A user study indicated that MARRS motions are perceived as more natural, smoother, and physically realistic by a significant majority of participants.
Limitations and Future Work
- Dataset Limitations: The current evaluation is limited to NTU120-AS and Chi3D-AS due to the lack of high-quality human reaction datasets. The authors suggest further exploration of physical constraints to address issues like foot sliding.
- Fine-Grained Motion Accuracy: The generation of very fine-grained motions, such as finger contacts, is still limited by dataset accuracy.
- Future Work: The paper suggests exploring more challenging aspects of human reaction synthesis and encourages the community to contribute to this area. The MARRS framework itself could be extended to handle more complex interactions and a larger number of interacting individuals.
Summary updated