Abstract
Accurate prediction of a vehicle's future state is critical for safe autonomous navigation. This paper introduces a deep learning framework for multi-horizon forecasting of steering angle and speed by fusing front-facing video with historical sensor data. By predicting multiple future time points simultaneously, the proposed models leverage visual context to anticipate road geometry and obstacles. Experimental results on the Udacity and Comma2k19 datasets demonstrate that visual features significantly improve forecasting accuracy, reducing error by over 30% compared to signal-only models.
Introduction
Most contemporary research in autonomous driving focuses on estimating the current state of a vehicle. However, safe driving is inherently a planning task; human drivers look ahead to anticipate road curvature and potential hazards. Current approaches often suffer from two major gaps: a lack of foresight and a failure to handle data imbalance. In real-world driving datasets, vehicles spend the vast majority of time driving straight. This "long-tailed" distribution means standard error metrics like Mean Absolute Error (MAE) can be misleadingly low even if a model fails during critical sharp turns.
Methodology
The researchers proposed a modular end-to-end architecture designed to fuse two distinct modalities: visual features and historical signal traces.
Feature Extraction
- Visual Features: A sequence of 10 video frames is processed using a lightweight 3D-CNN (MFNET-3D) to capture both spatial and temporal context from the road ahead.
- Signal Features: Historical traces of steering and speed are processed using 1D-CNNs to capture the vehicle's immediate motion trends.
Fusion and Prediction Architectures
The study explored three distinct fusion strategies for multi-horizon prediction:
- MH-IND-FC: Predicts each future timestamp individually using separate fully connected heads.
- MH-SIM-FC: Predicts all future horizons simultaneously through a single shared fully connected block.
- MH-SIM-LSTM: Employs an LSTM-based sequence-to-sequence approach. This proved most effective by capturing the temporal dependencies of the output trajectory.
Evaluation Protocol
Models were evaluated on the Udacity Driving Dataset and Comma2k19. To address the data imbalance issue, the authors introduced MAE@α, a metric that calculates error only for steering angles greater than a threshold α. This focuses evaluation on challenging maneuvers where safety is most at risk.
Key Insights and Results
The integration of vision transforms the model from a reactive system to a proactive one. While historical sensor data describes past behavior, the video feed serves as a "future sensor," revealing upcoming turns not yet present in the steering history.
Key findings include:
- Significant Error Reduction: Models incorporating vision achieved errors that were only 56.6% (Udacity) and 66.9% (Comma2k19) of the errors produced by signal-only models.
- Robustness in Edge Cases: Using the MAE@α metric, the MH-SIM-LSTM model maintained high accuracy even as steering angles became extreme, whereas non-vision models saw dramatic error spikes.
- Architectural Synergy: The LSTM-based fusion outperformed other methods by treating the forecast as a continuous, dependent trajectory rather than a set of independent points.
Conclusion and Future Work
This work demonstrates that visual context is indispensable for long-term vehicle state forecasting. While the current implementation uses a lightweight 3D-CNN, real-time processing on edge devices remains a challenge. Future research may explore Transformer-based architectures and the inclusion of additional modalities like GPS or LiDAR to further enhance forecasting reliability in diverse driving environments.
Summary updated