Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU

Junaid Ahmed Ansari, Brojeshwar Bhowmick

Abstract

We present a simple, fast, and light-weight RNN based framework for forecasting future locations of humans in first person monocular videos. The primary motivation for this work was to design a network which could accurately predict future trajectories at a very high rate on a CPU. Typical applications of such a system would be a social robot or a visual assistance system for all, as both cannot afford to have high compute power to avoid getting heavier, less power efficient, and costlier. In contrast to many previous methods which rely on multiple type of cues such as camera ego-motion or 2D pose of the human, we show that a carefully designed network model which relies solely on bounding boxes can not only perform better but also predicts trajectories at a very high rate while being quite low in size of approximately 17 MB. Specifically, we demonstrate that having an auto-encoder in the encoding phase of the past information and a regularizing layer in the end boosts the accuracy of predictions with negligible overhead. We experiment with three first person video datasets: CityWalks, FPL and JAAD. Our simple method trained on CityWalks surpasses the prediction accuracy of state-of-the-art method (STED) while being 9.6x faster on a CPU (STED runs on a GPU). To the best of our knowledge, we are the first to accurately forecast trajectories at a very high prediction rate of 78 trajectories per second on CPU.

The Challenge of First-Person Forecasting

Forecasting human movement in first-person videos (FPV) is inherently complex due to the "double motion" problem: the camera wearer is moving, and the observed subjects are also in motion. Existing state-of-the-art (SOTA) solutions often rely on a heavy mix of features, including optical flow, 2D/3D body poses, and camera ego-motion. While effective, these methods typically require significant GPU resources, making them impractical for portable devices like social robots or visual assistance wearables. This research demonstrates that a lean, bounding-box-only approach can achieve superior performance on standard CPU hardware.

Methods: A Lightweight Seq2Seq Architecture

The proposed framework utilizes a Sequence-to-Sequence (Seq2Seq) model built on Long Short-Term Memory (LSTM) units. The architecture is designed for efficiency and robustness through three key components:

  1. Encoder Block (Auto-encoder): Unlike standard encoders, this block is trained to reconstruct the input sequence of past bounding boxes. This forces the network to learn a more robust latent representation of motion dynamics.
  2. Decoder Block: This component predicts future velocities (changes in position and scale) rather than absolute coordinates, which improves stability and generalization across different resolutions.
  3. Trajectory Concatenation Layer: A non-learnable layer that integrates predicted velocities with the last known position, acting as a physical regularizer for the final trajectory.
Proposed RNN-based framework architecture featuring an auto-encoder block and velocity-predicting decoder.

The model operates on a simple 8-dimensional input vector per frame, capturing centroid coordinates, dimensions, and their temporal changes.

Experimental Setup and Datasets

The model's performance was validated across three distinct datasets:

  • CityWalks: A large-scale dataset featuring 358 sequences across 21 cities, used for primary training.
  • First Person Locomotion (FPL): Used to evaluate zero-shot transferability to unseen environments.
  • Joint Attention for Autonomous Driving (JAAD): Used to test performance after minimal (15%) fine-tuning in pedestrian-driver interaction scenarios.

Performance and Efficiency

The proposed model achieves a remarkable balance of speed and accuracy. It runs at 78 Trajectories Per Second (TPS) on a standard CPU, representing a 9.6x speedup over the GPU-dependent STED model. Despite its small size (~17 MB), it outperforms more complex models in accuracy.

Comparison of prediction accuracy (ADE/FDE) on the CityWalks dataset.

On the CityWalks benchmark, the model achieved an Average Displacement Error (ADE) of 21.61, significantly lower than the 26.0 achieved by STED. Furthermore, the model demonstrated strong generalization, performing on par with specialized models in zero-shot tests on the FPL dataset.

Conclusion

By prioritizing architectural simplicity and focusing on essential temporal dynamics, this work proves that high-performance human motion forecasting does not require specialized hardware or complex feature engineering. The resulting lightweight model provides a practical foundation for real-time applications in mobile robotics and assistive technology.

Summary updated

Abstract

We present a simple, fast, and light-weight RNN based framework for forecasting future locations of humans in first person monocular videos. The primary motivation for this work was to design a network which could accurately predict future trajectories at a very high rate on a CPU. Typical applications of such a system would be a social robot or a visual assistance system for all, as both cannot afford to have high compute power to avoid getting heavier, less power efficient, and costlier. In contrast to many previous methods which rely on multiple type of cues such as camera ego-motion or 2D pose of the human, we show that a carefully designed network model which relies solely on bounding boxes can not only perform better but also predicts trajectories at a very high rate while being quite low in size of approximately 17 MB. Specifically, we demonstrate that having an auto-encoder in the encoding phase of the past information and a regularizing layer in the end boosts the accuracy of predictions with negligible overhead. We experiment with three first person video datasets: CityWalks, FPL and JAAD. Our simple method trained on CityWalks surpasses the prediction accuracy of state-of-the-art method (STED) while being 9.6x faster on a CPU (STED runs on a GPU). We also demonstrate that our model can transfer zero-shot or after just 15% fine-tuning to other similar datasets and perform on par with the state-of-the-art methods on such datasets (FPL and DTP). To the best of our knowledge, we are the first to accurately forecast trajectories at a very high prediction rate of 78 trajectories per second on CPU.
BibTeX
@inproceedings{ansari2020simple,
  author = {Junaid Ahmed Ansari and Brojeshwar Bhowmick},
  title = {{Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=acYI1duUllm},
}
Figure