MARRS: Masked Autoregressive Unit-based Reaction Synthesis

Yabiao Wang, Shuo Wang, Jiangning Zhang, Jiafu Wu, Qing He, Yong Liu

TL;DR

This paper introduces MARRS, a novel framework for generating human reactions conditioned on another person's actions. MARRS addresses limitations of existing methods by using continuous representations, a unit-based approach for body and hands, and a novel mutual modulation mechanism to improve coordination.

What Problem Does This Paper Address?

Generating realistic human reactions to actions is a challenging task with applications in animation, gaming, and robotics. Current methods often rely on vector quantization (VQ), which can lead to information loss and codebook collapse. While autoregressive models and diffusion models have shown promise, they can be computationally expensive or struggle with fine-grained coordination between different body parts. Specifically, existing approaches often neglect the crucial aspect of mutual perception between distinct body units (like the torso and hands), leading to less coordinated and natural reactions.

Methods

MARRS tackles this by employing a two-stage approach:

  1. Unit-distinguished Motion Variational AutoEncoder (UD-VAE): This stage splits the whole-body motion into two distinct units: "body" and "hands." Each unit is then encoded independently using a VAE. This allows the model to learn representations for these parts separately, avoiding information loss associated with VQ.
  2. Masked Reaction Generation Model: This stage comprises several key components:
  3. Action-Conditioned Fusion (ACF): This module takes the active (actor's) motion and masked reactive (reactor's) tokens. It uses attention mechanisms to extract relevant information from the actor's motion and fuse it with the reactor's partially masked tokens.
  4. Mutual Unit Modulation (MUM): To address the lack of coordination between body and hands, MUM allows information from one unit to adaptively modulate the other. This bidirectional communication ensures that body and hand movements are synchronized and contextually appropriate.
  5. Diffusion MLPs: Instead of a single diffusion model for the entire motion, MARRS uses compact MLPs as noise predictors for each distinct unit (body and hands). This factorizes the problem, making optimization easier and improving learning efficiency. The diffusion loss is applied to model the probability distribution of each token individually.

Datasets

The paper evaluates MARRS on the following datasets:

  • NTU120-AS: A large-scale dataset for human action-reaction synthesis.
  • Chi3D-AS: Another dataset for human action-reaction synthesis.
  • Inter-X: A dataset specifically designed for human-human interaction.

Key Insights and Ideas

  • Unit Division is Crucial: Splitting the body into distinct units like "body" and "hands" significantly improves reaction synthesis, outperforming methods that treat the whole body as a single unit or use simpler divisions (e.g., upper/lower body).
  • Mutual Modulation for Coordination: The MUM mechanism, allowing body and hands to influence each other, is vital for generating natural and coordinated reactions. Bidirectional modulation (body influencing hands and vice-versa) yields the best results.
  • Continuous Representations Beat VQ: Avoiding vector quantization (VQ) and working with continuous representations directly addresses information loss and codebook collapse issues inherent in VQ-based methods.
  • Factorized Diffusion for Efficiency: Using separate diffusion models for body and hands units, conditioned on actor motion, simplifies the generation process and improves learning.

Key Results

MARRS demonstrates state-of-the-art performance across various metrics, including FID (Frechet Inception Distance), Acc (Accuracy of action recognition), Div (Diversity), and Multimod (Multimodality).

  • Quantitative: On NTU120-AS, MARRS achieved the best FID, Acc, and Multimod scores in both train-conditioned and test-conditioned settings. Similar superior performance was observed on Chi3D-AS and Inter-X datasets.
  • Qualitative: Visualizations show that MARRS produces more plausible body movements, relative positions, and natural hand gestures compared to existing methods like ReGenNet. The generated reactions are more synchronized and realistic.
  • User Study: A user study indicated that MARRS motions are perceived as more natural, smoother, and physically realistic by a significant majority of participants.

Limitations and Future Work

  • Dataset Limitations: The current evaluation is limited to NTU120-AS and Chi3D-AS due to the lack of high-quality human reaction datasets. The authors suggest further exploration of physical constraints to address issues like foot sliding.
  • Fine-Grained Motion Accuracy: The generation of very fine-grained motions, such as finger contacts, is still limited by dataset accuracy.
  • Future Work: The paper suggests exploring more challenging aspects of human reaction synthesis and encourages the community to contribute to this area. The MARRS framework itself could be extended to handle more complex interactions and a larger number of interacting individuals.

Summary updated

Abstract

This work aims at a challenging task: human action-reaction synthesis, i.e., generating human reactions conditioned on the action sequence of another person. Currently, autoregressive modeling approaches with vector quantization (VQ) have achieved remarkable performance in motion generation tasks. However, VQ has inherent disadvantages, including quantization information loss, low codebook utilization, etc. In addition, while dividing the body into separate units can be beneficial, the computational complexity needs to be considered. Also, the importance of mutual perception among units is often neglected. In this work, we propose MARRS, a novel framework designed to generate coordinated and fine-grained reaction motions using continuous representations. Initially, we present the Unit-distinguished Motion Variational AutoEncoder (UD-VAE), which segments the entire body into distinct body and hand units, encoding each independently. Subsequently, we propose Action-Conditioned Fusion (ACF), which involves randomly masking a subset of reactive tokens and extracting specific information about the body and hands from the active tokens. Furthermore, we introduce Mutual Unit Modulation (MUM) to facilitate interaction between body and hand units by using the information from one unit to adaptively modulate the other. Finally, for the diffusion model, we employ a compact MLP as a noise predictor for each distinct body unit and incorporate the diffusion loss to model the probability distribution of each token. Both quantitative and qualitative results demonstrate that our method achieves superior performance. Project page: https://aigc-explorer.github.io/MARRS/.
BibTeX
@article{wang2026marrs,
  author = {Yabiao Wang and Shuo Wang and Jiangning Zhang and Jiafu Wu and Qing He and Yong Liu},
  title = {{MARRS: Masked Autoregressive Unit-based Reaction Synthesis}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2026},
  doi = {10.1109/tvcg.2026.3675978},
}
Figure