Towards Robust Human Trajectory Prediction in Raw Videos

Rui Yu, Zihan Zhou

Introduction: The Reality of Raw Video

Human trajectory prediction is a critical component for autonomous systems, yet most research assumes access to "perfect" ground-truth past trajectories. In real-world applications like autonomous driving or robotics, systems must operate on raw video data. In this setting, the "past" is estimated by object detectors and trackers, which are prone to several types of errors that degrade prediction performance.

Common tracking errors in raw video: (a) noisy tracks, (b) missed targets, (c) spurious tracks, and (d) ID switches.

As shown in the figure above, raw video inputs often suffer from:

  • Noisy tracks: Jitter in bounding box placement creates erratic velocity estimates.
  • Missed targets: Occlusions cause pedestrians to disappear temporarily.
  • Spurious tracks: Background objects (e.g., shadows) are misidentified as people.
  • ID switches: Identities are swapped when pedestrians pass close to each other.

The Re-tracking Framework

To bridge the gap between imperfect tracking and robust prediction, the authors propose a standalone "re-tracking" module. This module acts as a corrective layer that can be integrated into any existing tracking-prediction pipeline.

The proposed pipeline integrates a re-tracking module between initial detection/tracking and final prediction.

The framework consists of three core innovations:

  1. Input Smoothing: The system applies the Holt-Winters method (double exponential smoothing) to instantaneous velocities. By smoothing these noisy inputs, the resulting trajectory predictions become significantly more stable.
  2. Redundant Predictions: Instead of a single prediction per track, the model runs at every tracked location. This redundancy allows the system to verify temporal consistency across multiple frames.
  3. Association via Prediction: This is the central shift in methodology. Traditional trackers associate detections based on spatial proximity. The re-tracking module instead associates them based on the similarity of their long-term future predictions. By modeling predicted locations as Gaussian distributions and using the Mahalanobis distance, the system can maintain identities even through long occlusions where spatial overlap fails.

Evaluation and Results

The method was rigorously tested on the Stanford Drone Dataset (SDD) and the WILDTRACK dataset. These benchmarks provide challenging bird's-eye and eye-level perspectives of crowded environments.

The re-tracking approach yielded substantial improvements:

  • Identity Stability: On the SDD dataset, identity switches were reduced by over 65%, dropping from 3,611 to 1,246.
  • Tracking Quality: Metrics such as Identity F1 (IDF1) improved from 36.0 to 44.9, while MOT Accuracy (MOTA) increased from 22.6 to 30.1.
  • Prediction Accuracy: The Average Displacement Error (ADE) was consistently lower than baseline methods across various recall thresholds.
  • Real-time Performance: The pipeline is computationally efficient, processing frames in approximately 0.004 seconds on standard hardware.

Conclusion

The key insight of this work is that prediction is not just an end goal, but a powerful tool for improving tracking. By enforcing long-term temporal consistency, the re-tracking module effectively filters out the noise inherent in raw video. While the current system is a downstream module, future work aims to develop jointly learned models that integrate detection, tracking, and prediction into a single, unified neural network.

Summary updated

Abstract

Human trajectory prediction has received increased attention lately due to its importance in applications such as autonomous vehicles and indoor robots. However, most existing methods make predictions based on human-labeled trajectories and ignore the errors and noises in detection and tracking. In this paper, we study the problem of human trajectory forecasting in raw videos, and show that the prediction accuracy can be severely affected by various types of tracking errors. Accordingly, we propose a simple yet effective strategy to correct the tracking failures by enforcing prediction consistency over time. The proposed "re-tracking" algorithm can be applied to any existing tracking and prediction pipelines. Experiments on public benchmark datasets demonstrate that the proposed method can improve both tracking and prediction performance in challenging real-world scenarios. The code and data are available at https://git.io/retracking-prediction.
BibTeX
@inproceedings{yu2021towards,
  author = {Rui Yu and Zihan Zhou},
  title = {{Towards Robust Human Trajectory Prediction in Raw Videos}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=0UjCn60CCab},
}
Figure