Social Diffusion: Long-term Multiple Human Motion Anticipation

Julian Tanke, Linguang Zhang, Amy Zhao, Chengcheng Tang, Yujun Cai, Lezi Wang, Po-Chen Wu, Juergen Gall, Cem Keskin

Abstract

Predicting long-term 3D human motion in social contexts remains a significant challenge due to the complex interdependencies between individuals and the tendency of traditional models to "freeze" or drift into implausible states over time. We present Social Diffusion (SDM), a generative framework that leverages diffusion models and social-aware aggregation to forecast realistic, multi-person interactions over extended durations (10+ seconds). By modeling motion as a denoising process conditioned on past observations, SDM maintains physical realism and social consistency, such as conversational turn-taking and role preservation, significantly outperforming existing state-of-the-art methods.

Introduction

Human motion is inherently social. Whether navigating a crowded room or engaging in a negotiation, our movements are continuously influenced by the actions and perceived intentions of others. While single-person motion prediction has seen substantial progress, multi-person anticipation often fails to capture these "social" nuances. Existing auto-regressive models frequently suffer from error accumulation, leading to static "mean poses" or socially nonsensical behaviors after just a few seconds.

Social Diffusion addresses these hurdles by moving away from step-by-step frame prediction. Instead, it treats motion anticipation as a generative task, producing entire sequences that respect the social roles and interaction patterns established in the observed past.

Methodology: Generative Denoising and Social Aggregation

The core of the Social Diffusion model is a Generative Denoising process. Unlike auto-regressive models that predict frame $N+1$ from frame $N$, SDM starts with Gaussian noise representing the entire future sequence and iteratively refines it into a coherent 3D motion path. This process is conditioned on a "prefix" (the observed past motion of all participants).

Social Diffusion predicts future multi-person motion (right) based on observed past trajectories (left).

To ensure that the generated motions are interdependent, the architecture incorporates a Social Aggregation function ($\Gamma$). This order-invariant component allows information to flow between all individuals in the scene during the denoising steps. The researchers found that even simple averaging across individual features provides a powerful inductive bias for maintaining social consistency. Furthermore, the fully convolutional nature of the temporal networks allows the model to scale "zero-shot" to larger groups than those seen during training.

Experimental Evaluation

The model was evaluated on standard benchmarks (MuPoTS-3D, 3DPW, CMU-Mocap) and a specialized Haggling Dataset. The latter features triadic interactions between a buyer and two sellers, providing a rigorous test for social role preservation.

Metrics for Social Realism

Standard metrics like Mean Per Joint Positional Error (MPJPE) often fail to quantify the "social" quality of a prediction. To address this, the authors introduced the Symbolic Social Cues (SSCP) Protocol. This protocol maps continuous motion into discrete social states (e.g., "speaking" vs. "listening"). By comparing the distribution of these states in the predicted future to ground truth data, the researchers could verify the model's "social logic."

Key Results

Social Diffusion demonstrates remarkable stability and accuracy over long horizons:

  • Accuracy: On the 3DPW dataset, SDM reduced MPJPE by over 54% for 3-second forecasts compared to previous state-of-the-art (SOTA) methods.
  • Long-Term Stability: While SOTA models often degrade after 4 seconds, SDM maintains high Normalized Directional Motion Similarity (NDMS) even 10 seconds into the future.
  • Role Preservation: In the Haggling dataset, SDM successfully "remembered" social roles. A buyer-detection network could identify the buyer in SDM's forecasts with ~89% accuracy, compared to near-random performance (~32%) from competing models.
The model captures complex social interactions, such as multiple participants engaging in conversation simultaneously or sequentially.

Discussion and Future Work

Social Diffusion represents a significant step toward socially-aware AI. By prioritizing the global structure of interaction over local frame-to-frame transitions, it avoids the "freeze" common in previous models.

However, limitations remain. The model currently struggles with absolute global positioning over very long durations and assumes a fixed number of participants. Future research will focus on "open-world" scenarios where individuals join or leave groups dynamically and integrating environmental constraints like furniture to enable more robust applications in social robotics and virtual reality.

Summary updated

Abstract

We propose Social Diffusion, a novel method for short-term and long-term forecasting of the motion of multiple persons as well as their social interactions. Jointly forecasting motions for multiple persons involved in social activities is inherently a challenging problem due to the interdependencies between individuals. In this work, we leverage a diffusion model conditioned on motion histories and causal temporal convolutional networks to forecast individually and contextually plausible motions for all participants. The contextual plausibility is achieved via an order-invariant aggregation function. As a second contribution, we design a new evaluation protocol that measures the plausibility of social interactions which we evaluate on the Haggling dataset, which features a challenging social activity where people are actively taking turns to talk and switching their attention. We evaluate our approach on four datasets for multi-person forecasting where our approach outperforms the state-of-the-art in terms of motion realism and contextual plausibility.
BibTeX
@inproceedings{tanke2023social,
  author = {Julian Tanke and Linguang Zhang and Amy Zhao and Chengcheng Tang and Yujun Cai and Lezi Wang and Po-Chen Wu and Juergen Gall and Cem Keskin},
  title = {{Social Diffusion: Long-term Multiple Human Motion Anticipation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00880},
}
Figure