Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU

Junaid Ahmed Ansari, Brojeshwar Bhowmick

This paper, titled "Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU," presents a compelling case for "less is more" in the world of computer vision. While many researchers are chasing accuracy by piling on complex features like optical flow and 3D pose estimation, Junaid Ahmed Ansari and Brojeshwar Bhowmick demonstrate that a lean, well-designed Recurrent Neural Network (RNN) can outperform the giants using nothing but simple bounding boxes.

TL;DR

The authors introduce a lightweight RNN-based framework that predicts future human trajectories in first-person videos using only bounding box data. By eschewing computationally expensive cues like optical flow or ego-motion, the model achieves state-of-the-art accuracy while running nearly 10x faster than existing methods on a standard CPU.

What Problem Does This Paper Address?

Forecasting where a person will move in a first-person video (FPV) is notoriously difficult. Unlike static surveillance footage, FPV involves a "double motion" problem: both the camera wearer and the observed subjects are moving.

Current state-of-the-art (SOTA) methods typically tackle this by feeding the network a buffet of information: camera ego-motion, 2D/3D body poses, and optical flow. While accurate, these methods are "compute-heavy" and usually require powerful GPUs. This is a dealbreaker for real-world applications like social robots or visual assistance wearables for the blind, which need to be lightweight, power-efficient, and low-cost. The authors set out to prove that you can achieve real-time forecasting on a modest CPU without sacrificing precision.

Methods

The proposed architecture is a Sequence-to-Sequence (Seq2Seq) model built primarily on Long Short-Term Memory (LSTM) units. It consists of three main components:

  1. Encoder Block (Auto-encoder): Unlike standard encoders that just compress data, this block functions as an auto-encoder. It processes the past $k$ frames of bounding box data and tries to reconstruct that same past sequence. This forces the network to learn a more robust latent representation of the "physics" of the observed motion.
  2. Decoder Block: This block takes the latent representation and predicts future velocities (changes in position and scale) rather than absolute coordinates.
  3. Trajectory Concatenation Layer: This is a non-learnable layer that integrates the predicted velocities with the last known position to produce the final future trajectory. The authors argue this acts as a regularizer, keeping the predictions physically plausible.

Crucially, the model only uses an 8-dimensional input vector per frame: $[cx, cy, w, h, \Delta cx, \Delta cy, \Delta w, \Delta h]$ (centroid coordinates, width, height, and their respective changes).

Datasets

The model was evaluated on three diverse first-person datasets:

  • CityWalks: 358 video sequences across 21 European cities. This served as the primary training and benchmark set.
  • First Person Locomotion (FPL): Used to test "zero-shot" transferability (applying the model to a new dataset without retraining).
  • Joint Attention for Autonomous Driving (JAAD): A dataset focused on pedestrian-driver interactions, used to test the model's performance after minimal (15%) fine-tuning.

Key Insights and Ideas

  • Bounding Boxes are Enough: The most surprising finding is that high-level features like optical flow or 2D pose are often redundant for trajectory forecasting if the temporal dynamics are captured correctly.
  • The Power of Reconstruction: By forcing the encoder to act as an auto-encoder, the model learns a much richer summary of the past. This "hidden" task significantly boosts the accuracy of the "future" task.
  • Velocity over Position: Predicting the change in state (velocity) is more stable than predicting absolute future coordinates. It allows the model to generalize better across different camera resolutions and frame rates.
  • CPU-First Design: By keeping the model tiny (~17 MB), the authors achieve a massive speedup, proving that deep learning doesn't always require a "GPU-or-bust" mentality.

Key Results

  • Speed: The model runs at 78 Trajectories Per Second (TPS) on a standard CPU. This is 9.6x faster than the STED model, which requires a GPU to run.
  • Accuracy: On the CityWalks dataset, the model achieved an Average Displacement Error (ADE) of 21.61, significantly better than STED's 26.0.
  • Efficiency: The entire model is only 17.4 MB, making it ideal for mobile or embedded deployment.
  • Generalization: In zero-shot tests on the FPL dataset, the model performed on par with specialized models that had access to much more complex features.

Limitations and Future Work

While the results are impressive, the authors acknowledge some hurdles:

  • Dependency on Detection: The model relies on an external object detector (like YOLO or Mask-RCNN) to provide the initial bounding boxes. If the detector fails or jitters, the forecast will suffer.
  • Social Context: The current model treats each human independently. It does not explicitly model "social pooling" or how pedestrians avoid each other in a crowd.
  • Future Directions: The authors suggest that integrating this lightweight forecaster into an end-to-end navigation system for AR glasses or low-cost robotics is the logical next step.

By stripping away the complexity, Ansari and Bhowmick have provided a blueprint for practical, real-time AI that can run on the hardware we actually carry in our pockets.

Summary updated

Abstract

We present a simple, fast, and light-weight RNN based framework for forecasting future locations of humans in first person monocular videos. The primary motivation for this work was to design a network which could accurately predict future trajectories at a very high rate on a CPU. Typical applications of such a system would be a social robot or a visual assistance system "for all", as both cannot afford to have high compute power to avoid getting heavier, less power efficient, and costlier. In contrast to many previous methods which rely on multiple type of cues such as camera ego-motion or 2D pose of the human, we show that a carefully designed network model which relies solely on bounding boxes can not only perform better but also predicts trajectories at a very high rate while being quite low in size of approximately 17 MB. Specifically, we demonstrate that having an auto-encoder in the encoding phase of the past information and a regularizing layer in the end boosts the accuracy of predictions with negligible overhead. We experiment with three first person video datasets: CityWalks, FPL and JAAD. Our simple method trained on CityWalks surpasses the prediction accuracy of state-of-the-art method (STED) while being 9.6x faster on a CPU (STED runs on a GPU). We also demonstrate that our model can transfer zero-shot or after just 15% fine-tuning to other similar datasets and perform on par with the state-of-the-art methods on such datasets (FPL and DTP). To the best of our knowledge, we are the first to accurately forecast trajectories at a very high prediction rate of 78 trajectories per second on CPU.
BibTeX
@inproceedings{ansari2020simple,
  author = {Junaid Ahmed Ansari and Brojeshwar Bhowmick},
  title = {{Simple means Faster: Real-Time Human Motion Forecasting in Monocular First Person Videos on CPU}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=acYI1duUllm},
}

Loading related content...

Figure