Abstract
We tackle the problem of Human Locomotion Forecasting, a task for jointly predicting the spatial positions of several keypoints on the human body in the near future under an egocentric setting. In contrast to previous work that aims to solve either pose prediction or trajectory forecasting in isolation, we propose a framework to unify the two problems and address the practically useful task of pedestrian locomotion prediction in the wild. Our method disentangles overall pedestrian motion into global (trajectory) and local (pose keypoint movements) subparts. To overcome the scarcity of annotated egocentric video datasets, we utilize state-of-the-art models to generate noisy supervision and propose robust forecasting models that can learn from this data. Our approach achieves state-of-the-art results for human locomotion prediction in egocentric views.
What Problem Does This Paper Address?
Predicting where a pedestrian is going and how they will move is critical for self-driving cars and social robots to avoid collisions and plan safe paths. Most previous research treats trajectory forecasting (where the person goes) and pose forecasting (how their body moves) as separate problems.
Doing this from an egocentric perspective (a camera mounted on a moving car) adds layers of complexity:
- Camera Motion: The car's own movement (egomotion) distorts the perceived motion of the pedestrian.
- Depth Perception: As pedestrians move toward or away from the car, their scale changes, making it hard to distinguish between actual movement and perspective shifts.
- Data Scarcity: There are very few datasets that provide "ground truth" labels for human poses, depth, and vehicle motion in real-world driving scenarios.
Methods
The authors propose a multi-stage pipeline designed to work with "noisy" supervision rather than perfect human labels. The core innovation lies in the disentanglement of motion into distinct streams.
- Noisy Supervision Generation: Instead of manual labels, the system uses off-the-shelf models to estimate poses (OpenPose), depth (SuperDepth), and camera motion from raw video.
- Pose Completion: Because pose detectors often miss joints due to occlusion, a symmetric autoencoder is used to "fill in the blanks" and denoise the detected skeletons.
- Disentanglement: The locomotion is split into two streams:
- Global Stream: Tracks the "neck" joint as a proxy for the person's overall trajectory, incorporating egomotion and depth signals.
- Local Stream: Tracks how every other joint moves relative to the neck, capturing the specific limb movements.
- Quasi-RNN (QRNN) Backbone: The model uses QRNNs in an Encoder-Recurrent-Decoder architecture, which are faster and often more effective than standard LSTMs for sequence tasks.
- Recombination: The predicted global and local movements are merged back together to create the final forecasted locomotion.
Key Insights and Ideas
- Divide and Conquer: Predicting raw joint coordinates is unnecessarily hard. By separating the "where" (global) from the "how" (local), the model can learn simpler, more specialized patterns.
- Embracing Noise: The paper proves that you don't need perfect human-annotated data to train high-performing models. A robust architecture can filter out the "garbage" from automated detectors.
- Domain-Specific Signals Matter: For egocentric views, simply looking at 2D coordinates isn't enough. Integrating depth and egomotion allows the model to understand if a pedestrian is actually walking faster or if the car is just accelerating toward them.
Key Results
The authors evaluated their model using Keypoint Displacement Error (KDE), which measures the average distance between predicted and actual joint positions on the JAAD (Joint Attention in Autonomous Driving) dataset.
- Quantitative: Their method achieved a KDE of 10.9, significantly outperforming standard sequence models like GRU-ED (23.8) and specialized models like Structural RNN (31.7).
- Ablation Studies: Disentanglement was found to be the single most important factor; without it, the error nearly doubled.
- Qualitative: Visualizations show the model accurately capturing complex motions, such as the swinging of legs during a stride, even when the camera is moving.
Limitations and Future Work
While the model handles noise well, it is still limited by the quality of the off-the-shelf detectors. Furthermore, the paper focuses on short-term forecasting (up to 1 second). Long-term forecasting remains a challenge due to the multimodality of human intent and the accumulation of errors in camera motion estimation over time.
Summary updated