Forecasting Time-to-Collision from Monocular Video: Feasibility, Dataset, and Challenges

Aashi Manglik, Xinshuo Weng, Eshed Ohn-Bar, Kris M. Kitani

Abstract

Predicting the exact time-to-collision (TTC) is a fundamental requirement for safe autonomous navigation, particularly for assistive robots interacting with pedestrians. This paper presents a deep learning-based regression approach that forecasts TTC using only monocular video. By introducing a large-scale dataset of over 13,000 near-collision events and evaluating a multi-stream 2D Convolutional Neural Network (CNN), the authors demonstrate that robots can learn to predict collisions up to 6 seconds in advance. The findings suggest that implicit motion tracking via 2D CNNs outperforms both traditional physics-based models and complex 3D spatio-temporal architectures.

Introduction

Automated collision avoidance is a cornerstone of mobile robotics. For assistive devices, such as suitcase-shaped robots designed to guide visually impaired users, the ability to anticipate and react to human movement is critical. Traditional approaches typically rely on active sensors like LIDAR or RGB-D cameras to track pedestrians and extrapolate their trajectories using physics-based models. However, these methods are often hampered by sensor limitations—LIDAR can struggle in bright outdoor light, and physics-based extrapolation is highly sensitive to tracking noise. A minor jitter in a bounding box can lead to massive spikes in estimated velocity, resulting in unreliable collision warnings.

While recent deep learning research has explored monocular collision avoidance, many models treat the problem as a binary classification task (e.g., "collision imminent" vs. "safe"). While useful for simple obstacle avoidance in UAVs, binary labels lack the granularity needed for smooth path planning in crowded environments. This paper addresses this gap by framing TTC prediction as a continuous regression task, aiming to estimate the precise time until a near-collision occurs in milliseconds.

Methodology

The core of the proposed approach is the transition from explicit tracking to implicit motion learning. Instead of detecting a person and calculating their velocity through successive frames, the model learns to infer motion directly from raw pixel data.

Multi-stream Architecture

To capture temporal dynamics, the network processes a sequence of $N$ consecutive RGB frames. The authors propose a Multi-stream VGG architecture:

  1. Each frame in the temporal window is passed independently through a shared VGG-16 backbone to extract spatial features.
  2. These features are concatenated in their original temporal order.
  3. The resulting flattened vector is processed through fully connected layers to output a single scalar representing the TTC.

The study also evaluates an Inflated 3D ConvNet (I3D) to determine if explicit 3D spatio-temporal kernels provide an advantage over the 2D multi-stream approach.

The Near-Collision Dataset

A significant hurdle in TTC research is the lack of diverse, labeled data. Collecting real-world collision data is inherently dangerous and impractical. To solve this, the authors curated a novel dataset of 13,685 egocentric indoor video segments featuring near-collision events.

  • Definition of Near-Collision: A "near-collision" is defined as a pedestrian entering a 1-meter radius around the mobile platform.
  • Automated Labeling: To generate ground-truth TTC values without manual annotation, the collection rig utilized a stereo camera and LIDAR. Faster R-CNN was used to detect pedestrians, and LIDAR point clouds were projected onto these detections to calculate precise distances.
  • Scale: This dataset is significantly larger and more specialized for pedestrian interactions than previous benchmarks like the UAV crashing dataset or DroNet.

Experimental Results

The performance of the models was measured using Mean Absolute Error (MAE) in seconds.

Comparative Performance

The Multi-stream VGG model using a 6-frame input (0.5 seconds of history) emerged as the top performer:

  • Multi-stream VGG (6-frame): 0.753s MAE
  • I3D (3D CNN): 0.811s MAE
  • DroNet: 1.09s MAE
  • Explicit Tracking + Linear Model: 1.05s MAE
  • Constant Baseline: 1.38s MAE

The 2D CNN approach not only achieved higher accuracy but also produced more temporally consistent predictions compared to the 3D CNN, which exhibited higher output fluctuations.

Forecast Horizon

The difficulty of prediction scales with the temporal distance to the event. For collisions occurring within 1 second, the model achieved an MAE of approximately 0.60 seconds. For events 5–6 seconds into the future, the error increased to 1.83 seconds. This indicates that while long-range forecasting is possible, the model is most reliable for immediate tactical adjustments.

Discussion and Insights

The Temporal "Sweet Spot"

One key finding is that more history is not always better. While a single frame is insufficient for motion estimation, feeding the network more than 0.5 seconds of history (6 frames at 10 fps) did not yield significant performance gains. This suggests that the immediate trajectory of a pedestrian is the most informative signal for short-term TTC forecasting.

Limitations and Future Work

Despite its success, the model faces challenges with directional ambiguity. If a pedestrian is walking in the same direction as the robot (moving away), the network may still predict a collision because it detects a person in the field of view. Additionally, because the dataset was collected indoors at specific walking speeds (0.2–1.5 m/s), further work is needed to ensure generalization to outdoor environments.

The authors suggest that future models should output both a mean TTC and a variance. This variance would act as a confidence indicator, allowing the robot's planning system to identify when a prediction is unreliable and should be treated with caution.

Conclusion

This research demonstrates the feasibility of forecasting TTC from monocular video using deep regression models. By bypassing explicit tracking and leveraging a large-scale, automatically labeled dataset, the proposed Multi-stream VGG architecture provides a robust alternative to traditional sensor-heavy methods, enabling smoother and more proactive robot navigation.

Summary updated

Abstract

We explore the possibility of using a single monocular camera to forecast the time to collision between a suitcase-shaped robot being pushed by its user and other nearby pedestrians. We develop a purely image-based deep learning approach that directly estimates the time to collision without the need of relying on explicit geometric depth estimates or velocity information to predict future collisions. While previous work has focused on detecting immediate collision in the context of navigating Unmanned Aerial Vehicles, the detection was limited to a binary variable (i.e., collision or no collision). We propose a more fine-grained approach to collision forecasting by predicting the exact time to collision in terms of milliseconds, which is more helpful for collision avoidance in the context of dynamic path planning. To evaluate our method, we have collected a novel dataset of over 13,000 indoor video segments each showing a trajectory of at least one person ending in a close proximity (a near collision) with the camera mounted on a mobile suitcase-shaped platform. Using this dataset, we do extensive experimentation on different temporal windows as input using an exhaustive list of state-of-the-art convolutional neural networks (CNNs). Our results show that our proposed multi-stream CNN is the best model for predicting time to near-collision. The average prediction error of our time to near-collision is 0.75 seconds across the test videos. The project webpage can be found at https://aashi7.github.io/NearCollision.html.
BibTeX
@inproceedings{manglik2019forecasting,
  author = {Aashi Manglik and Xinshuo Weng and Eshed Ohn-Bar and Kris M. Kitani},
  title = {{Forecasting Time-to-Collision from Monocular Video: Feasibility, Dataset, and Challenges}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967730},
}
Figure