Social Motion Prediction with Cognitive Hierarchies

Wentao Zhu, Jason Qin, Yuke Lou, Hang Ye, Xiaoxuan Ma, Hai Ci, Yizhou Wang

Introduction

Predicting human motion in social environments is a fundamental challenge in computer vision and robotics. While existing models perform adequately in "casual" settings—such as pedestrians navigating a sidewalk—they often fail in "strategic" environments where movements are governed by specific objectives and adversarial interactions. In these contexts, an agent's next move is not merely a function of their current velocity, but a result of complex social action planning.

Researchers from Peking University have addressed these limitations by introducing a novel framework that treats social motion prediction as a Multi-Agent Reinforcement Learning (MARL) problem. By leveraging a Cognitive Hierarchy (CH) framework and a new high-intensity dataset, Wusi, this research shifts the focus from simple trajectory regression to modeling the recursive reasoning inherent in human competition.

The Wusi Dataset: A Benchmark for Strategic Motion

A significant hurdle in motion prediction research has been the lack of high-quality, goal-oriented data. Most multi-person datasets feature low-intensity interactions where movements are somewhat stochastic. To bridge this gap, the authors released the Wusi Basketball Training Dataset.

Wusi consists of 60,000 frames (approximately 40.3 minutes) of 3D motion capture data featuring five participants in 3-on-2 "no-dribble" basketball drills. This specific constraint is crucial; by removing dribbling, the players must rely entirely on strategic positioning and passing to achieve their goals. Compared to traditional benchmarks like CMU-Mocap or 3DPW, Wusi exhibits significantly higher motion intensity (measured by average joint velocity) and greater pose diversity. This dataset provides a rigorous environment where every movement has a clear underlying "why," making the prediction task both more challenging and more meaningful.

Methodology: Cognitive Hierarchies and Imitation Learning

The core of the proposed framework is the integration of behavioral economics into motion modeling. The researchers move beyond standard sequence-to-sequence models by implementing a Cognitive Hierarchy (CH) framework.

Recursive Reasoning

The CH framework models agents at varying levels of strategic depth:

  • Level-0: Agents act reactively based on the current state without considering others' intentions.
  • Level-1: Agents anticipate that others are Level-0 actors and plan counter-moves accordingly.
  • Level-k: Agents recursively anticipate the actions of Level-$(k-1)$ agents.

By stacking these policy networks, the model explicitly simulates the "I think that you think..." logic common in competitive sports. This allows the system to predict complex behaviors like "fake passes" or defensive rotations that require understanding an opponent's intent.

Learning the Policy

To train these agents, the authors utilize a combination of Behavioral Cloning (BC) and Generative Adversarial Imitation Learning (GAIL). While BC provides training efficiency, GAIL acts as a critic to ensure the generated motions remain physically plausible. This hybrid approach prevents common artifacts in motion synthesis, such as unnatural limb twisting, ensuring that the predicted trajectories are both strategically sound and visually realistic.

Key Results and Insights

The framework was evaluated against state-of-the-art (SOTA) models, including MRT and SoMoFormer.

Quantitative Performance

The proposed method achieved the lowest Mean Per Joint Position Error (MPJPE) across nearly all time horizons. A standout result was observed in long-term root trajectory prediction (1000ms), where the model outperformed the nearest SOTA by a significant margin—124.0mm compared to 131.2mm. Ablation studies further confirmed that increasing the strategic depth ($k$) directly correlates with a drop in prediction error, validating the importance of recursive reasoning.

Qualitative Tactical Awareness

Beyond metrics, the model demonstrated superior tactical understanding. Qualitative visualizations showed the model correctly predicting a defender "switching" targets to cover an open offensive player. Standard models, which typically project current velocity forward, failed to capture this high-level tactical shift, highlighting the advantage of modeling the underlying "game" rather than just the motion.

Conclusion and Future Directions

This research demonstrates that strategic constraints are a feature, not a bug, in motion prediction. By modeling the "why" behind human movement through Cognitive Hierarchies, the authors have set a new standard for multi-agent interaction modeling.

While the framework is highly effective in structured environments like basketball, future work remains to be done regarding its generalization to less "rational" or more chaotic social settings. Additionally, incorporating explicit social roles (e.g., specific positions in a team) could further refine the specificity of the predictions. As it stands, the Wusi dataset and the CH framework provide a robust foundation for the next generation of social AI.

Summary updated

BibTeX
@inproceedings{zhu2023social,
  author = {Wentao Zhu and Jason Qin and Yuke Lou and Hang Ye and Xiaoxuan Ma and Hai Ci and Yizhou Wang},
  title = {{Social Motion Prediction with Cognitive Hierarchies}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=lRu0dN7BY6},
}
Figure