Peeking into the Future: Predicting Future Person Activities and Locations in Videos

Junwei Liang

Abstract

Predicting human movement is a cornerstone of autonomous systems, yet traditional methods often treat pedestrians as simple points in space. This paper introduces Next, a multi-task learning framework that jointly predicts a person’s future path and their intended activity. By integrating rich visual cues—including body pose, scene semantics, and person-object interactions—the model moves beyond simple trajectory forecasting to understand human intention. Evaluated on the ActEV/VIRAT and ETH/UCY benchmarks, the system demonstrates state-of-the-art performance, proving that activity recognition and path prediction are mutually beneficial tasks.

Introduction

Accurately forecasting where a person will walk is essential for the safe operation of self-driving cars, social robots, and intelligent surveillance systems. Historically, trajectory prediction has focused on modeling people as "particles" moving on a 2D plane. However, human movement is inherently goal-oriented. For instance, a person carrying a box toward a vehicle is likely to stop at the trunk rather than continue in a straight line. This research addresses the gap between low-level path tracking and high-level activity understanding, positing that predicting where someone is going requires understanding what they intend to do.

The Next Framework

The authors propose Next, an end-to-end multi-task learning architecture designed to capture the nuances of human behavior and environmental context.

Overview of the Next framework, illustrating the integration of person behavior, interaction modules, and the multi-task output for trajectory and activity prediction.

The system is composed of several specialized modules:

  • Person Behavior Module: This module extracts visual features of a person’s appearance using RoIAlign and captures body language through 2D keypoint estimation.
  • Person Interaction Module: It encodes the relationship between the person and their surroundings, specifically looking at Person-Scene interactions (e.g., sidewalk vs. grass) and Person-Object interactions (e.g., proximity to vehicles).
  • Trajectory Generator: An LSTM-based decoder employs a "focal attention" mechanism to dynamically weigh visual features over time, generating the future $(x,y)$ path.

Activity Prediction and the Manhattan Grid

A core innovation of this work is the use of a Manhattan Grid for activity prediction. To localize where a future activity will occur, the scene is divided into a multi-scale grid.

The Manhattan Grid mechanism for multi-scale activity location prediction, utilizing classification and regression to anchor trajectories.

The model performs two simultaneous tasks on this grid:

  1. Classification: Identifying which grid cell will contain the future activity.
  2. Regression: Determining the precise coordinates within that cell.

This auxiliary task serves as a powerful regularizer, "anchoring" the predicted trajectory to a logical destination and preventing the path from drifting aimlessly.

Experimental Results

The model was rigorously tested on two major benchmarks:

  • ActEV/VIRAT: A large-scale dataset with 29 activity classes. The Next model achieved an Average Displacement Error (ADE) of 17.99, a significant improvement over the previous state-of-the-art Social GAN (ADE 23.11).
  • ETH & UCY: Standard pedestrian benchmarks where the model outperformed existing methods even without explicit activity labels, demonstrating the robustness of its path-finding capabilities.

Key insights from the results include the fact that joint modeling is a "win-win": activity prediction informs the path, and the path provides context for the activity. Furthermore, visual cues like body pose and scene semantics are critical for distinguishing between multiple plausible future trajectories.

Conclusion and Future Work

The Next framework demonstrates that understanding human intention is vital for accurate trajectory forecasting. While the model currently focuses on a short-term horizon (approx. 5 seconds) and routine activities, it sets a new standard for multi-task prediction in complex scenes. Future research will look toward longer-term horizons, more complex social interactions among groups, and the ethical implications of deploying activity-predictive systems in public spaces.

Summary updated

Abstract

Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in many applications. Motivated by this idea, this paper studies predicting a pedestrian's future path jointly with future activities. We propose an end-to-end, multi-task learning system utilizing rich visual features about human behavioral information and interaction with their surroundings. To facilitate the training, the network is learned with an auxiliary task of predicting future location in which the activity will happen. Experimental results demonstrate our state-of-the-art performance over two public benchmarks on future trajectory prediction. Moreover, our method is able to produce meaningful future activity prediction in addition to the path. The result provides the first empirical evidence that joint modeling of paths and activities benefits future path prediction.
BibTeX
@inproceedings{liang2019peeking,
  author = {Junwei Liang},
  title = {{Peeking into the Future: Predicting Future Person Activities and Locations in Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00587},
}
Figure