Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning

Jaewoo Jeong, Daehee Park, Kuk-Jin Yoon

Introduction

Predicting human movement in crowded environments is a fundamental challenge for autonomous systems, such as self-driving cars and social robots. While existing human pose forecasting models have made strides, they often struggle with two critical limitations: short prediction horizons (typically under 3 seconds) and computational complexity that scales poorly with the number of agents and joints. Furthermore, many current benchmarks rely on synthetic data that fails to capture the nuances of real-world social interactions, such as collision avoidance or group coordination.

To address these challenges, researchers from KAIST have introduced Trajectory2Pose (T2P), a framework designed for long-term (up to 5 seconds) 3D pose forecasting of multiple agents. By decoupling global navigation from local body mechanics, T2P achieves state-of-the-art performance while maintaining computational efficiency.

The Trajectory2Pose (T2P) Framework

The core innovation of T2P is a coarse-to-fine strategy that decomposes the complex task of multi-agent pose prediction into two manageable steps: global trajectory forecasting and local pose forecasting.

Overview of the trajectory-conditioned pose forecasting approach, illustrating the transition from global intent (trajectories) to local intent (detailed poses).

1. Global Trajectory Forecasting (Coarse)

The model first predicts the future path of each person’s root joint (the hip). This "global trajectory" represents the agent's high-level intent—where they are going and how they navigate around others. By focusing on trajectories first, the model can efficiently capture social interactions without the overhead of modeling every individual joint.

2. Local Pose Forecasting (Fine)

Once the global paths are established, the model predicts the movement of limbs relative to the root joint. This step is conditioned on the predicted trajectory; for instance, if a trajectory indicates a sharp turn, the local pose model generates the corresponding leaning and leg movements.

Technical Architecture

The T2P architecture utilizes several specialized modules to handle the interplay between trajectories and poses:

  • Pose Encoder: Extracts features from historical 3D poses using Discrete Cosine Transform (DCT) and multi-head self-attention.
  • Trajectory Encoder: Comprises a Trajectory Module for social interaction modeling and a Traj-Pose Module that fuses trajectory and pose embeddings via graph attention.
  • Decoders: Separate decoders generate the final global trajectories and local pose sequences, which are then combined to produce the full 3D output.
Detailed architecture of the Trajectory2Pose (T2P) model, highlighting the Trajectory and Pose Encoders/Decoders and the interaction-aware modules.

By modeling interactions at the trajectory level ($O(T \cdot N^2)$) rather than the joint level ($O(T \cdot N^2 \cdot J^2)$), T2P is significantly more scalable for crowded scenes.

Datasets and Evaluation

The researchers evaluated T2P on standard benchmarks like CMU-Mocap (UMPM) and 3DPW. However, to truly test the model's capabilities in complex environments, they introduced JRDB-GlobMultiPose (JRDB-GMP).

Derived from the JRDB robotics dataset, JRDB-GMP features up to 24 agents in real-world campus settings. Unlike synthetic datasets, it contains authentic human behaviors—people walking in groups, avoiding obstacles, and performing diverse actions like tying shoes or riding scooters.

Key Results and Insights

  • Superior Accuracy: T2P consistently outperformed existing methods across all benchmarks. On the challenging JRDB-GMP dataset, it achieved a 10% improvement in Final Distance Error (FDE) over previous state-of-the-art models.
  • Reciprocal Feedback: A key finding was that while global intent informs local motion, the reverse is also true. A person's body language (local pose) often provides early cues about their next directional change (global trajectory).
  • Realistic Motion: Qualitative analysis showed that T2P produces more physically plausible motions, avoiding common artifacts like "sliding" feet or inter-agent collisions that plague less sophisticated models.

Conclusion and Future Directions

Trajectory2Pose represents a significant advancement in multi-agent pose forecasting by effectively balancing global social navigation with local body dynamics. While the current model relies on 3D poses "lifted" from 2D data, future work aims to incorporate environmental context (like maps and obstacles) and explore stochastic modeling techniques to better capture the inherent uncertainty of human behavior.

Summary updated

Abstract

Human pose forecasting garners attention for its diverse applications. However, challenges in modeling the multi-modal nature of human motion and intricate interactions among agents persist, particularly with longer timescales and more agents. In this paper, we propose an interaction-aware trajectory-conditioned long-term multi-agent human pose forecasting model, utilizing a coarse-to-fine prediction approach: multi-modal global trajectories are initially forecasted, followed by respective local pose forecasts conditioned on each mode. In doing so, our Trajectory2Pose model introduces a graph-based agent-wise interaction module for a reciprocal forecast of local motion-conditioned global trajectory and trajectory-conditioned local pose. Our model effectively handles the multi-modality of human motion and the complexity of long-term multi-agent interactions, improving performance in complex environments. Furthermore, we address the lack of long-term (6s+) multi-agent (5+) datasets by constructing a new dataset from real-world images and 2D annotations, enabling a comprehensive evaluation of our proposed model. State-of-the-art prediction performance on both complex and simpler datasets confirms the generalized effectiveness of our method. The code is available at https://github.com/Jaewoo97/T2P.
BibTeX
@inproceedings{jeong2024multi,
  author = {Jaewoo Jeong and Daehee Park and Kuk-Jin Yoon},
  title = {{Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure