Vision-Guided Forecasting - Visual Context for Multi-Horizon Time Series Forecasting

Eitan Kosman, Dotan Di CastroarXiv 2021 · arXiv

Abstract

Accurate prediction of a vehicle's future state is critical for safe autonomous navigation. This paper introduces a deep learning framework for multi-horizon forecasting of steering angle and speed by fusing front-facing video with historical sensor data. By predicting multiple future time points simultaneously, the proposed models leverage visual context to anticipate road geometry and obstacles. Experimental results on the Udacity and Comma2k19 datasets demonstrate that visual features significantly improve forecasting accuracy, reducing error by over 30% compared to signal-only models.

Introduction

Most contemporary research in autonomous driving focuses on estimating the current state of a vehicle. However, safe driving is inherently a planning task; human drivers look ahead to anticipate road curvature and potential hazards. Current approaches often suffer from two major gaps: a lack of foresight and a failure to handle data imbalance. In real-world driving datasets, vehicles spend the vast majority of time driving straight. This "long-tailed" distribution means standard error metrics like Mean Absolute Error (MAE) can be misleadingly low even if a model fails during critical sharp turns.

Distribution of steering angles in driving datasets, highlighting the long-tailed nature of the data.

Methodology

The researchers proposed a modular end-to-end architecture designed to fuse two distinct modalities: visual features and historical signal traces.

Feature Extraction

  • Visual Features: A sequence of 10 video frames is processed using a lightweight 3D-CNN (MFNET-3D) to capture both spatial and temporal context from the road ahead.
  • Signal Features: Historical traces of steering and speed are processed using 1D-CNNs to capture the vehicle's immediate motion trends.

Fusion and Prediction Architectures

The study explored three distinct fusion strategies for multi-horizon prediction:

  1. MH-IND-FC: Predicts each future timestamp individually using separate fully connected heads.
  2. MH-SIM-FC: Predicts all future horizons simultaneously through a single shared fully connected block.
  3. MH-SIM-LSTM: Employs an LSTM-based sequence-to-sequence approach. This proved most effective by capturing the temporal dependencies of the output trajectory.
The proposed multi-horizon forecasting architecture fusing visual and signal features.

Evaluation Protocol

Models were evaluated on the Udacity Driving Dataset and Comma2k19. To address the data imbalance issue, the authors introduced MAE@α, a metric that calculates error only for steering angles greater than a threshold α. This focuses evaluation on challenging maneuvers where safety is most at risk.

Key Insights and Results

The integration of vision transforms the model from a reactive system to a proactive one. While historical sensor data describes past behavior, the video feed serves as a "future sensor," revealing upcoming turns not yet present in the steering history.

Key findings include:

  • Significant Error Reduction: Models incorporating vision achieved errors that were only 56.6% (Udacity) and 66.9% (Comma2k19) of the errors produced by signal-only models.
  • Robustness in Edge Cases: Using the MAE@α metric, the MH-SIM-LSTM model maintained high accuracy even as steering angles became extreme, whereas non-vision models saw dramatic error spikes.
  • Architectural Synergy: The LSTM-based fusion outperformed other methods by treating the forecast as a continuous, dependent trajectory rather than a set of independent points.

Conclusion and Future Work

This work demonstrates that visual context is indispensable for long-term vehicle state forecasting. While the current implementation uses a lightweight 3D-CNN, real-time processing on edge devices remains a challenge. Future research may explore Transformer-based architectures and the inclusion of additional modalities like GPS or LiDAR to further enhance forecasting reliability in diverse driving environments.

Summary updated

Abstract

Autonomous driving gained huge traction in recent years, due to its potential to change the way we commute. Much effort has been put into trying to estimate the state of a vehicle. Meanwhile, learning to forecast the state of a vehicle ahead introduces new capabilities, such as predicting dangerous situations. Moreover, forecasting brings new supervision opportunities by learning to predict richer a context, expressed by multiple horizons. Intuitively, a video stream originated from a front-facing camera is necessary because it encodes information about the upcoming road. Besides, historical traces of the vehicle's states give more context. In this paper, we tackle multi-horizon forecasting of vehicle states by fusing the two modalities. We design and experiment with 3 end-to-end architectures that exploit 3D convolutions for visual features extraction and 1D convolutions for features extraction from speed and steering angle traces. To demonstrate the effectiveness of our method, we perform extensive experiments on two publicly available real-world datasets, Comma2k19 and the Udacity challenge. We show that we are able to forecast a vehicle's state to various horizons, while outperforming the current state-of-the-art results on the related task of driving state estimation. We examine the contribution of vision features, and find that a model fed with vision features achieves an error that is 56.6% and 66.9% of the error of a model that doesn't use those features, on the Udacity and Comma2k19 datasets respectively.
BibTeX
@misc{kosman2021vision,
  author = {Eitan Kosman and Dotan Di Castro},
  title = {{Vision-Guided Forecasting - Visual Context for Multi-Horizon Time Series Forecasting}},
  year = {2021},
  eprint = {2107.12674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.12674},
}
Figure