Abstract
Predicting human movement is a cornerstone of autonomous systems, yet traditional methods often treat pedestrians as simple points in space. This paper introduces Next, a multi-task learning framework that jointly predicts a person’s future path and their intended activity. By integrating rich visual cues—including body pose, scene semantics, and person-object interactions—the model moves beyond simple trajectory forecasting to understand human intention. Evaluated on the ActEV/VIRAT and ETH/UCY benchmarks, the system demonstrates state-of-the-art performance, proving that activity recognition and path prediction are mutually beneficial tasks.
Introduction
Accurately forecasting where a person will walk is essential for the safe operation of self-driving cars, social robots, and intelligent surveillance systems. Historically, trajectory prediction has focused on modeling people as "particles" moving on a 2D plane. However, human movement is inherently goal-oriented. For instance, a person carrying a box toward a vehicle is likely to stop at the trunk rather than continue in a straight line. This research addresses the gap between low-level path tracking and high-level activity understanding, positing that predicting where someone is going requires understanding what they intend to do.
The Next Framework
The authors propose Next, an end-to-end multi-task learning architecture designed to capture the nuances of human behavior and environmental context.
The system is composed of several specialized modules:
- Person Behavior Module: This module extracts visual features of a person’s appearance using RoIAlign and captures body language through 2D keypoint estimation.
- Person Interaction Module: It encodes the relationship between the person and their surroundings, specifically looking at Person-Scene interactions (e.g., sidewalk vs. grass) and Person-Object interactions (e.g., proximity to vehicles).
- Trajectory Generator: An LSTM-based decoder employs a "focal attention" mechanism to dynamically weigh visual features over time, generating the future $(x,y)$ path.
Activity Prediction and the Manhattan Grid
A core innovation of this work is the use of a Manhattan Grid for activity prediction. To localize where a future activity will occur, the scene is divided into a multi-scale grid.
The model performs two simultaneous tasks on this grid:
- Classification: Identifying which grid cell will contain the future activity.
- Regression: Determining the precise coordinates within that cell.
This auxiliary task serves as a powerful regularizer, "anchoring" the predicted trajectory to a logical destination and preventing the path from drifting aimlessly.
Experimental Results
The model was rigorously tested on two major benchmarks:
- ActEV/VIRAT: A large-scale dataset with 29 activity classes. The Next model achieved an Average Displacement Error (ADE) of 17.99, a significant improvement over the previous state-of-the-art Social GAN (ADE 23.11).
- ETH & UCY: Standard pedestrian benchmarks where the model outperformed existing methods even without explicit activity labels, demonstrating the robustness of its path-finding capabilities.
Key insights from the results include the fact that joint modeling is a "win-win": activity prediction informs the path, and the path provides context for the activity. Furthermore, visual cues like body pose and scene semantics are critical for distinguishing between multiple plausible future trajectories.
Conclusion and Future Work
The Next framework demonstrates that understanding human intention is vital for accurate trajectory forecasting. While the model currently focuses on a short-term horizon (approx. 5 seconds) and routine activities, it sets a new standard for multi-task prediction in complex scenes. Future research will look toward longer-term horizons, more complex social interactions among groups, and the ethical implications of deploying activity-predictive systems in public spaces.
Summary updated