Abstract
Predicting the exact time-to-collision (TTC) is a fundamental requirement for safe autonomous navigation, particularly for assistive robots interacting with pedestrians. This paper presents a deep learning-based regression approach that forecasts TTC using only monocular video. By introducing a large-scale dataset of over 13,000 near-collision events and evaluating a multi-stream 2D Convolutional Neural Network (CNN), the authors demonstrate that robots can learn to predict collisions up to 6 seconds in advance. The findings suggest that implicit motion tracking via 2D CNNs outperforms both traditional physics-based models and complex 3D spatio-temporal architectures.
Introduction
Automated collision avoidance is a cornerstone of mobile robotics. For assistive devices, such as suitcase-shaped robots designed to guide visually impaired users, the ability to anticipate and react to human movement is critical. Traditional approaches typically rely on active sensors like LIDAR or RGB-D cameras to track pedestrians and extrapolate their trajectories using physics-based models. However, these methods are often hampered by sensor limitations—LIDAR can struggle in bright outdoor light, and physics-based extrapolation is highly sensitive to tracking noise. A minor jitter in a bounding box can lead to massive spikes in estimated velocity, resulting in unreliable collision warnings.
While recent deep learning research has explored monocular collision avoidance, many models treat the problem as a binary classification task (e.g., "collision imminent" vs. "safe"). While useful for simple obstacle avoidance in UAVs, binary labels lack the granularity needed for smooth path planning in crowded environments. This paper addresses this gap by framing TTC prediction as a continuous regression task, aiming to estimate the precise time until a near-collision occurs in milliseconds.
Methodology
The core of the proposed approach is the transition from explicit tracking to implicit motion learning. Instead of detecting a person and calculating their velocity through successive frames, the model learns to infer motion directly from raw pixel data.
Multi-stream Architecture
To capture temporal dynamics, the network processes a sequence of $N$ consecutive RGB frames. The authors propose a Multi-stream VGG architecture:
- Each frame in the temporal window is passed independently through a shared VGG-16 backbone to extract spatial features.
- These features are concatenated in their original temporal order.
- The resulting flattened vector is processed through fully connected layers to output a single scalar representing the TTC.
The study also evaluates an Inflated 3D ConvNet (I3D) to determine if explicit 3D spatio-temporal kernels provide an advantage over the 2D multi-stream approach.
The Near-Collision Dataset
A significant hurdle in TTC research is the lack of diverse, labeled data. Collecting real-world collision data is inherently dangerous and impractical. To solve this, the authors curated a novel dataset of 13,685 egocentric indoor video segments featuring near-collision events.
- Definition of Near-Collision: A "near-collision" is defined as a pedestrian entering a 1-meter radius around the mobile platform.
- Automated Labeling: To generate ground-truth TTC values without manual annotation, the collection rig utilized a stereo camera and LIDAR. Faster R-CNN was used to detect pedestrians, and LIDAR point clouds were projected onto these detections to calculate precise distances.
- Scale: This dataset is significantly larger and more specialized for pedestrian interactions than previous benchmarks like the UAV crashing dataset or DroNet.
Experimental Results
The performance of the models was measured using Mean Absolute Error (MAE) in seconds.
Comparative Performance
The Multi-stream VGG model using a 6-frame input (0.5 seconds of history) emerged as the top performer:
- Multi-stream VGG (6-frame): 0.753s MAE
- I3D (3D CNN): 0.811s MAE
- DroNet: 1.09s MAE
- Explicit Tracking + Linear Model: 1.05s MAE
- Constant Baseline: 1.38s MAE
The 2D CNN approach not only achieved higher accuracy but also produced more temporally consistent predictions compared to the 3D CNN, which exhibited higher output fluctuations.
Forecast Horizon
The difficulty of prediction scales with the temporal distance to the event. For collisions occurring within 1 second, the model achieved an MAE of approximately 0.60 seconds. For events 5–6 seconds into the future, the error increased to 1.83 seconds. This indicates that while long-range forecasting is possible, the model is most reliable for immediate tactical adjustments.
Discussion and Insights
The Temporal "Sweet Spot"
One key finding is that more history is not always better. While a single frame is insufficient for motion estimation, feeding the network more than 0.5 seconds of history (6 frames at 10 fps) did not yield significant performance gains. This suggests that the immediate trajectory of a pedestrian is the most informative signal for short-term TTC forecasting.
Limitations and Future Work
Despite its success, the model faces challenges with directional ambiguity. If a pedestrian is walking in the same direction as the robot (moving away), the network may still predict a collision because it detects a person in the field of view. Additionally, because the dataset was collected indoors at specific walking speeds (0.2–1.5 m/s), further work is needed to ensure generalization to outdoor environments.
The authors suggest that future models should output both a mean TTC and a variance. This variance would act as a confidence indicator, allowing the robot's planning system to identify when a prediction is unreliable and should be treated with caution.
Conclusion
This research demonstrates the feasibility of forecasting TTC from monocular video using deep regression models. By bypassing explicit tracking and leveraging a large-scale, automatically labeled dataset, the proposed Multi-stream VGG architecture provides a robust alternative to traditional sensor-heavy methods, enabling smoother and more proactive robot navigation.
Summary updated