MARRS: Masked Autoregressive Unit-based Reaction Synthesis

Jiangning Zhang (张江宁), Y. B. Wang, S. Wang, J. N. Zhang, J. F. Wu, Q. D. He, C. C. Fu, C. J. Wang, Y. Liu

Introduction

Synthesizing realistic human reactions to the actions of others—such as flinching from a punch or reaching for a handshake—is critical for believable animation and interactive gaming. Current approaches often struggle with a "Goldilocks" problem. Discrete models, such as those based on Vector Quantization (VQ-VAE), suffer from information loss and "codebook collapse," leading to repetitive motions. Conversely, pure diffusion models offer high quality but are computationally expensive and slow. Furthermore, most models treat the human body as a monolithic block, failing to capture the high-frequency nuances of hand gestures compared to torso movements.

MARRS (Masked Autoregressive Unit-based Reaction Synthesis) addresses these challenges by employing a continuous autoregressive framework that treats the body and hands as distinct but coordinated units.

Methodology

The MARRS framework consists of two primary stages designed to preserve motion detail and ensure efficient inference.

Unit-distinguished Motion VAE (UD-VAE)

To avoid the pitfalls of discrete tokens, MARRS utilizes a Unit-distinguished Motion Variational AutoEncoder. This component splits the skeleton into two specific units: the Body and the Hands. Each unit is encoded independently into a continuous latent space, allowing the model to capture the specific nuances of intricate hand gestures separately from larger body shifts. This "divide and modulate" strategy is central to preserving fine-grained details often lost in global encodings.

Masked Reaction Generation

The core of MARRS is a masked autoregressive transformer that predicts reaction tokens. This model introduces three key innovations:

  • Action-Conditioned Fusion (ACF): Employs cross-attention to allow reactor tokens to extract relevant information from the actor's motion.
  • Adaptive Unit Modulation (AUM): Ensures spatial and temporal consistency between the body and hands. AUM allows one unit to adaptively modulate the other via scale and shift parameters, facilitating "mutual perception" between body parts.
  • Diffusion MLP: Instead of heavy transformer-based denoisers, MARRS uses a compact 3-layer MLP as a noise predictor for each token. This drastically reduces computational overhead during the iterative generation process.
The MARRS architecture featuring UD-VAE and the Masked Reaction Generation Model with ACF and AUM modules.

Experimental Setup

The model was evaluated on two benchmark datasets specifically tailored for action-reaction synthesis:

  • NTU120-AS: A subset of the NTU RGB+D 120 dataset focusing on 26 interaction types (e.g., hitting, shaking hands).
  • Chi3D-AS: A subset of the Chi3D dataset providing high-quality 3D ground truth for close-contact interactions.

Key Results

MARRS achieves state-of-the-art performance across multiple quantitative metrics and qualitative user studies.

Motion Quality and Generalization

On the NTU120-AS dataset, MARRS achieved a Frechet Inception Distance (FID) of 0.36 in train-conditioned settings, a significant improvement over the previous state-of-the-art, ReGenNet (0.90). In test-conditioned settings (reacting to unseen motions), MARRS maintained its lead with an FID of 9.31 versus ReGenNet's 11.00.

Quantitative performance comparison on the NTU120-AS dataset, showing MARRS outperforming existing methods across various metrics.

Efficiency and User Preference

The "MARRS-Tiny" variant demonstrates exceptional efficiency, with an inference time of only 0.039s per sequence. In subjective evaluations, 76% of participants preferred MARRS-generated motions for their naturalness, while 79% rated them as more physically realistic than the baseline.

Conclusion

MARRS demonstrates that a continuous, unit-based autoregressive approach significantly improves the quality and speed of human reaction synthesis. By splitting the body into functional units and using lightweight MLPs for diffusion, the model achieves high-fidelity results suitable for interactive applications. While challenges like foot sliding and precise contact accuracy remain, MARRS sets a new benchmark for realistic multi-human interaction modeling.

Summary updated

BibTeX
@article{2026marrs,
  author = {Jiangning Zhang (张江宁) and Y. B. Wang and S. Wang and J. N. Zhang and J. F. Wu and Q. D. He and C. C. Fu and C. J. Wang and Y. Liu},
  title = {{MARRS: Masked Autoregressive Unit-based Reaction Synthesis}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2026},
  doi = {10.1109/tvcg.2026.3675978},
}
Figure