HUMOF: Human Motion Forecasting in Interactive Social Scenes

Caiyi Sun, Yujing Sun, Xiao Han, Zemin Yang, Jiawei Liu, Xinge Zhu, Siu Ming Yiu, Yuexin Ma

Abstract

Predicting human motion in complex social environments requires a deep understanding of both human-human and human-scene interactions. HUMOF (HUman MOtion Forecasting) is a novel framework that addresses this by employing hierarchical representations and a coarse-to-fine reasoning strategy in both spatial and frequency domains. By focusing on interactive distances rather than semantic labels and utilizing adaptive frequency rescaling, HUMOF achieves state-of-the-art accuracy across four major benchmarks, producing motions that are both precise and physically plausible.

Introduction

Human movement is rarely an isolated event; it is a dynamic response to the surrounding context. In social settings, our trajectories are shaped by interactions: avoiding collisions with others, reaching for objects, or navigating around furniture. Existing motion forecasting models often struggle because they either focus solely on individual kinematics or treat the environment as a static, labeled background. HUMOF overcomes these limitations by modeling the intricate relationships between multiple humans and their 3D environment as the primary drivers of motion.

Methods

The HUMOF architecture is built upon two foundational pillars designed to capture the complexity of social movement:

Hierarchical Interaction Representation

HUMOF decomposes interactions into two distinct streams:

  • Human-Human Interaction (HHI): This module features a "self-encoding" branch for independent motion and a "relation-encoding" branch that explicitly calculates distances between joints of different individuals.
  • Human-Scene Interaction (HSI): Utilizing PointNet++ style set abstractions, the model generates a multi-scale representation of the 3D point cloud. It calculates "interactive distances" between the person and the scene at various granularities, from the global room layout to fine-grained surface details.

Coarse-to-Fine Interaction Reasoning

The prediction engine uses a Transformer-based module to process information progressively across two dimensions:

  • Spatial Perspective: High-level global context is injected into early layers, while low-level geometric details are introduced later to refine the motion.
  • Frequency Perspective (Adaptive DCT Rescaling): Human motion is represented in the frequency domain using the Discrete Cosine Transform (DCT). HUMOF employs a learnable mechanism to suppress high-frequency jitter in early reasoning stages, allowing the model to establish a stable trajectory before refining the nuanced details of the motion.
The HUMOF architecture featuring hierarchical HHI and HSI representations and the coarse-to-fine reasoning module.

Key Results and Insights

HUMOF was evaluated on four diverse datasets: HIK (kitchen interactions), HOI-M3 (real-world scenes), GTA-IM (synthetic interactions), and HUMANISE (language-conditioned motion).

Performance and Plausibility

HUMOF consistently outperformed previous state-of-the-art methods such as SAST and MutualDistance. On the HIK dataset, it achieved a mean path error of 180.7mm, improving upon the previous best of 189.0mm. Beyond quantitative metrics, HUMOF demonstrates superior physical plausibility. While baseline models often predict "impossible" motions—such as legs clipping through floors or furniture—HUMOF's reliance on interactive distances ensures that predicted paths respect the physical boundaries of the scene.

Qualitative comparison showing HUMOF (Ours) producing physically plausible paths compared to baseline methods that suffer from collisions and clipping.

Core Innovations

  • Distance as a Universal Language: By prioritizing geometric distances over semantic labels (e.g., "chair"), the model becomes more robust to diverse environments where perfect labels may not be available.
  • Frequency-Aware Reasoning: Adaptive DCT Rescaling mimics human perception by prioritizing general directionality before fine-tuning specific limb movements, effectively reducing temporal noise.

Conclusion

HUMOF represents a significant step forward in social motion forecasting by treating movement as a multi-scale dialogue between the individual and their environment. While currently optimized for scenarios with rich interactions, its efficiency (43ms inference time) and scalability make it a promising foundation for real-time applications in autonomous driving and human-robot interaction.

Summary updated

BibTeX
@inproceedings{sun2026humof,
  author = {Caiyi Sun and Yujing Sun and Xiao Han and Zemin Yang and Jiawei Liu and Xinge Zhu and Siu Ming Yiu and Yuexin Ma},
  title = {{HUMOF: Human Motion Forecasting in Interactive Social Scenes}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=INy8guZqrm},
}
Figure