Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision

Karttikeya Mangalam, Ehsan Adeli, Kuan-Hui Lee, Adrien Gaidon, Juan Carlos Niebles

Abstract

Predicting pedestrian behavior is critical for the safety and efficiency of autonomous driving systems. This paper introduces a framework that jointly forecasts a pedestrian's future trajectory and body pose from an egocentric perspective. By disentangling global movement from local limb dynamics and utilizing a specialized pose-completion module, the proposed method achieves state-of-the-art results even when trained on noisy, machine-generated labels.

Introduction: The Egocentric Challenge

Predicting pedestrian movement is a fundamental challenge for autonomous systems. Traditional research often separates trajectory forecasting (2D map positioning) from pose forecasting (limb movement). However, in real-world driving scenarios, these elements are deeply intertwined. For instance, a pedestrian's posture often signals an imminent change in direction or speed.

Figure 1: Pedestrian locomotion forecasting from an egocentric perspective, illustrating the integration of pose and trajectory prediction.

Capturing this data from a moving vehicle (egocentric view) introduces significant noise due to camera motion (egomotion) and frequent occlusions. Furthermore, large-scale datasets with manual joint-level annotations are scarce, necessitating models that can learn from imperfect supervision.

Methodology: A Divide and Conquer Approach

The authors propose a three-stage pipeline to forecast "locomotion"—the combination of pose and trajectory.

Figure 2: The proposed pipeline, featuring pose estimation, completion, and the disentangled Global/Local RNN architecture.

1. Noisy Supervision and Pose Completion

To overcome the lack of manual labels, the framework uses off-the-shelf tools like OpenPose to generate initial keypoints. A Symmetric Autoencoder (Pose Completion module) is then employed to denoise these labels and fill in missing joints caused by occlusion.

2. Disentanglement of Dynamics

The core innovation lies in splitting motion into two distinct streams:

  • Global Stream: Tracks the "root" joint (the neck) to capture the overall path. This stream incorporates external signals like monocular depth and camera egomotion.
  • Local Stream: Tracks all other joints relative to the neck, capturing the kinematics of walking or arm swinging.

3. Hierarchical Forecasting

The model utilizes Quasi-Recurrent Neural Networks (QRNNs) for forecasting. QRNNs offer better parallelization and faster training speeds than standard LSTMs while effectively handling the temporal dependencies of human motion.

Experimental Results

The framework was evaluated on the JAAD (Joint Attention in Autonomous Driving) dataset, augmented with monocular depth (SuperDepth) and unsupervised egomotion estimation.

Performance Metrics

The primary metric used is Keypoint Displacement Error (KDE), measuring the average pixel distance between predicted and actual joint locations.

Model KDE (Lower is Better)
Structural RNN 31.7
GRU-ED 23.8
Proposed Method 10.9

Key Findings

  • Disentanglement is Essential: Ablation studies showed that removing the disentanglement module caused the KDE to jump from 10.9 to 18.2.
  • Robustness to Noise: The pose completion module successfully "cleans" machine-generated labels, proving that high-quality human annotations are not strictly necessary for training effective models.
  • Efficiency: The QRNN-based architecture trained faster (825 ms/batch) than LSTM-based alternatives (886 ms/batch) with superior convergence.

Conclusion and Future Directions

This research demonstrates that separating global pathing from local kinematics significantly simplifies the task of locomotion forecasting. While the current model relies on pre-existing bounding boxes and produces 2D outputs, it sets a strong foundation for future work in social disentanglement—modeling how multiple agents interact—and extending predictions into full 3D space.

Summary updated

Abstract

We tackle the problem of Human Locomotion Forecasting, a task for jointly predicting the spatial positions of several keypoints on human body in the near future under an egocentric setting. In contrast to the previous work that aims to solve either the task of pose prediction or trajectory forecasting in isolation, we propose a framework to unify these two problems and address the practically useful task of pedestrian locomotion prediction in the wild. Among the major challenges in solving this task is the scarcity of annotated egocentric video datasets with dense annotations for pose, depth, or egomotion. To surmount this difficulty, we use state-of-the-art models to generate (noisy) annotations and propose robust forecasting models that can learn from this noisy supervision. We present a method to disentangle the overall pedestrian motion into easier to learn subparts by uti-lizing a pose completion and a decomposition module. The completion module fills in the missing key-point annotations and the decomposition module breaks the cleaned locomotion down to global (trajectory) and local (pose keypoint movements). Further, with Quasi RNN as our backbone, we propose a novel hierarchical trajectory forecasting network that utilizes low-level vision domain specific signals like egomotion and depth to predict the global trajectory. Our method leads to state-of-the-art results for the prediction of human locomotion in the egocentric view.
BibTeX
@inproceedings{mangalam2020disentangling,
  author = {Karttikeya Mangalam and Ehsan Adeli and Kuan-Hui Lee and Adrien Gaidon and Juan Carlos Niebles},
  title = {{Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093350},
}
Figure