Abstract
Predicting human motion in complex social environments requires a deep understanding of both human-human and human-scene interactions. HUMOF (HUman MOtion Forecasting) is a novel framework that addresses this by employing hierarchical representations and a coarse-to-fine reasoning strategy in both spatial and frequency domains. By focusing on interactive distances rather than semantic labels and utilizing adaptive frequency rescaling, HUMOF achieves state-of-the-art accuracy across four major benchmarks, producing motions that are both precise and physically plausible.
Introduction
Human movement is rarely an isolated event; it is a dynamic response to the surrounding context. In social settings, our trajectories are shaped by interactions: avoiding collisions with others, reaching for objects, or navigating around furniture. Existing motion forecasting models often struggle because they either focus solely on individual kinematics or treat the environment as a static, labeled background. HUMOF overcomes these limitations by modeling the intricate relationships between multiple humans and their 3D environment as the primary drivers of motion.
Methods
The HUMOF architecture is built upon two foundational pillars designed to capture the complexity of social movement:
Hierarchical Interaction Representation
HUMOF decomposes interactions into two distinct streams:
- Human-Human Interaction (HHI): This module features a "self-encoding" branch for independent motion and a "relation-encoding" branch that explicitly calculates distances between joints of different individuals.
- Human-Scene Interaction (HSI): Utilizing PointNet++ style set abstractions, the model generates a multi-scale representation of the 3D point cloud. It calculates "interactive distances" between the person and the scene at various granularities, from the global room layout to fine-grained surface details.
Coarse-to-Fine Interaction Reasoning
The prediction engine uses a Transformer-based module to process information progressively across two dimensions:
- Spatial Perspective: High-level global context is injected into early layers, while low-level geometric details are introduced later to refine the motion.
- Frequency Perspective (Adaptive DCT Rescaling): Human motion is represented in the frequency domain using the Discrete Cosine Transform (DCT). HUMOF employs a learnable mechanism to suppress high-frequency jitter in early reasoning stages, allowing the model to establish a stable trajectory before refining the nuanced details of the motion.
Key Results and Insights
HUMOF was evaluated on four diverse datasets: HIK (kitchen interactions), HOI-M3 (real-world scenes), GTA-IM (synthetic interactions), and HUMANISE (language-conditioned motion).
Performance and Plausibility
HUMOF consistently outperformed previous state-of-the-art methods such as SAST and MutualDistance. On the HIK dataset, it achieved a mean path error of 180.7mm, improving upon the previous best of 189.0mm. Beyond quantitative metrics, HUMOF demonstrates superior physical plausibility. While baseline models often predict "impossible" motions—such as legs clipping through floors or furniture—HUMOF's reliance on interactive distances ensures that predicted paths respect the physical boundaries of the scene.
Core Innovations
- Distance as a Universal Language: By prioritizing geometric distances over semantic labels (e.g., "chair"), the model becomes more robust to diverse environments where perfect labels may not be available.
- Frequency-Aware Reasoning: Adaptive DCT Rescaling mimics human perception by prioritizing general directionality before fine-tuning specific limb movements, effectively reducing temporal noise.
Conclusion
HUMOF represents a significant step forward in social motion forecasting by treating movement as a multi-scale dialogue between the individual and their environment. While currently optimized for scenarios with rich interactions, its efficiency (43ms inference time) and scalability make it a promising foundation for real-time applications in autonomous driving and human-robot interaction.
Summary updated