Introduction
Synthesizing realistic human reactions to the actions of others—such as flinching from a punch or reaching for a handshake—is critical for believable animation and interactive gaming. Current approaches often struggle with a "Goldilocks" problem. Discrete models, such as those based on Vector Quantization (VQ-VAE), suffer from information loss and "codebook collapse," leading to repetitive motions. Conversely, pure diffusion models offer high quality but are computationally expensive and slow. Furthermore, most models treat the human body as a monolithic block, failing to capture the high-frequency nuances of hand gestures compared to torso movements.
MARRS (Masked Autoregressive Unit-based Reaction Synthesis) addresses these challenges by employing a continuous autoregressive framework that treats the body and hands as distinct but coordinated units.
Methodology
The MARRS framework consists of two primary stages designed to preserve motion detail and ensure efficient inference.
Unit-distinguished Motion VAE (UD-VAE)
To avoid the pitfalls of discrete tokens, MARRS utilizes a Unit-distinguished Motion Variational AutoEncoder. This component splits the skeleton into two specific units: the Body and the Hands. Each unit is encoded independently into a continuous latent space, allowing the model to capture the specific nuances of intricate hand gestures separately from larger body shifts. This "divide and modulate" strategy is central to preserving fine-grained details often lost in global encodings.
Masked Reaction Generation
The core of MARRS is a masked autoregressive transformer that predicts reaction tokens. This model introduces three key innovations:
- Action-Conditioned Fusion (ACF): Employs cross-attention to allow reactor tokens to extract relevant information from the actor's motion.
- Adaptive Unit Modulation (AUM): Ensures spatial and temporal consistency between the body and hands. AUM allows one unit to adaptively modulate the other via scale and shift parameters, facilitating "mutual perception" between body parts.
- Diffusion MLP: Instead of heavy transformer-based denoisers, MARRS uses a compact 3-layer MLP as a noise predictor for each token. This drastically reduces computational overhead during the iterative generation process.
Experimental Setup
The model was evaluated on two benchmark datasets specifically tailored for action-reaction synthesis:
- NTU120-AS: A subset of the NTU RGB+D 120 dataset focusing on 26 interaction types (e.g., hitting, shaking hands).
- Chi3D-AS: A subset of the Chi3D dataset providing high-quality 3D ground truth for close-contact interactions.
Key Results
MARRS achieves state-of-the-art performance across multiple quantitative metrics and qualitative user studies.
Motion Quality and Generalization
On the NTU120-AS dataset, MARRS achieved a Frechet Inception Distance (FID) of 0.36 in train-conditioned settings, a significant improvement over the previous state-of-the-art, ReGenNet (0.90). In test-conditioned settings (reacting to unseen motions), MARRS maintained its lead with an FID of 9.31 versus ReGenNet's 11.00.
Efficiency and User Preference
The "MARRS-Tiny" variant demonstrates exceptional efficiency, with an inference time of only 0.039s per sequence. In subjective evaluations, 76% of participants preferred MARRS-generated motions for their naturalness, while 79% rated them as more physically realistic than the baseline.
Conclusion
MARRS demonstrates that a continuous, unit-based autoregressive approach significantly improves the quality and speed of human reaction synthesis. By splitting the body into functional units and using lightweight MLPs for diffusion, the model achieves high-fidelity results suitable for interactive applications. While challenges like foot sliding and precise contact accuracy remain, MARRS sets a new benchmark for realistic multi-human interaction modeling.
Summary updated