Future Near-Collision Prediction from Monocular Video: Feasibility, Dataset, and Challenges.

Aashi Manglik, Xinshuo Weng, Eshed Ohn-Bar, Kris Kitani

Abstract

This paper presents a novel image-based deep learning approach for forecasting the exact time to a near-collision using monocular video. By defining a near-collision as a person entering a 1-meter radius around a mobile robot, the authors develop a multi-stream VGG network that processes temporal sequences of frames. To train and evaluate the model, a new dataset of over 13,000 indoor egocentric video segments was collected using a mobile suitcase robot equipped with LIDAR and stereo cameras for ground truth annotation. Experimental results show that the proposed architecture achieves a Mean Absolute Error (MAE) of 0.75 seconds, outperforming traditional tracking-based methods and 3D convolutional networks.

Introduction

Automated collision avoidance is a cornerstone of mobile robotics, particularly for assistive devices like smart suitcases designed for the visually impaired. Traditional systems rely on expensive sensor suites (LIDAR, RGB-D) to track pedestrians in 3D space and extrapolate trajectories. However, these physics-based models are highly sensitive to sensor noise and tracking failures. Recent monocular approaches often simplify the task into a binary classification of "collision" vs. "no collision." This work addresses a more granular predictive challenge: forecasting the precise time-to-collision (TTC) up to 6 seconds in advance.

Illustration of the time-to-near-collision concept, where a near-collision is defined as a pedestrian entering a 1-meter radius of the robot.

Methods

The authors frame TTC forecasting as a regression problem. To capture both spatial features and motion dynamics, they propose a Multi-stream VGG architecture.

The Multi-stream VGG architecture concatenates spatial features from a sequence of N RGB frames to predict the time-to-collision.

The network processes a sequence of recent frames (typically a 0.5-second window) independently through a VGG-16 backbone. The resulting spatial features are flattened and concatenated, preserving temporal order, before being passed through fully connected layers to regress a single TTC value. This approach avoids the complexity and potential instability of 3D convolutions (e.g., I3D) while providing smoother, more consistent predictions.

The Near-Collision Dataset

A significant contribution of this work is the Near-Collision Dataset, comprising 13,685 egocentric video sequences recorded at 10 fps in various indoor environments. The data was collected using a suitcase-shaped robot prototype. While the model uses only monocular RGB images at inference, ground truth TTC was established during training using a calibrated LIDAR and stereo camera setup. LIDAR point clouds were projected into 2D space to accurately detect when a pedestrian breached the 1-meter safety radius.

Results and Discussion

The Multi-stream VGG model achieved a Mean Absolute Error (MAE) of 0.753 seconds, significantly better than the I3D baseline (0.811s) and traditional tracking-plus-linear-velocity models (1.055s). Key findings include:

  • Temporal Window: A history of 6 frames (0.5 seconds) provides the optimal balance between motion context and computational efficiency.
  • Implicit Tracking: The model successfully learns to anticipate collisions directly from pixels, bypassing the error-prone step of explicit pedestrian detection and data association.
  • Horizon Sensitivity: Prediction accuracy naturally decreases as the forecast horizon increases, with MAE rising from ~0.60s for immediate collisions (0-1s) to ~1.84s for distant ones (5-6s).

Conclusion

This research demonstrates the feasibility of precise time-to-collision forecasting from monocular video. While the model shows promise, it currently struggles with pedestrians moving away from the robot and may require further adaptation for outdoor environments. Future work aims to incorporate uncertainty estimation by outputting both a mean and variance for the TTC, enhancing the reliability of downstream path-planning algorithms.

Summary updated

BibTeX
@misc{manglik2019future,
  author = {Aashi Manglik and Xinshuo Weng and Eshed Ohn-Bar and Kris Kitani},
  title = {{Future Near-Collision Prediction from Monocular Video: Feasibility, Dataset, and Challenges.}},
  year = {2019},
  eprint = {1903.09102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.09102},
}
Figure