End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

Simon Hecker, Dengxin Dai, Luc Van Gool

TL;DR

This paper tackles the challenge of building more robust end-to-end driving models by moving beyond single front-facing cameras. They introduce a new dataset and a deep learning architecture that integrates surround-view camera data and route planner information to predict steering angle and speed, showing significant improvements over single-camera approaches, especially in complex urban scenarios.

What Problem Does This Paper Address?

Current end-to-end driving models often rely on a single front-facing camera, which provides a limited view of the environment. This limitation makes it difficult for models to learn safe and effective driving behaviors, particularly in scenarios requiring awareness of surroundings (e.g., intersections, lane changes) or when navigating complex routes. Human drivers, in contrast, utilize a 360-degree view (mirrors) and an internal "mental map" for navigation. The paper aims to bridge this gap by developing driving models that incorporate richer sensory input, mimicking human drivers' use of comprehensive visual information and route planning.

Methods

The core of the paper is a novel deep learning architecture designed for end-to-end driving. This architecture takes as input:

  • Surround-view camera data: Videos from eight cameras providing a 360-degree view of the vehicle's surroundings. These are processed by ResNet34 (pre-trained on ImageNet) for feature extraction.
  • Route planner information: Two representations are used:
  • TomTom Map: Rendered video sequences from the TomTom Go Mobile app.
  • OpenStreetMap (OSM): A sequence of GPS coordinates representing the planned route.
  • Historical vehicle states: Previous steering angles and speeds.

These features are fused and fed into a multi-layer LSTM network to capture temporal dependencies. The fused information is then passed through fully-connected layers to predict the future steering angle and vehicle speed. The authors highlight that they also experimented with a setting that excludes historical driving maneuvers, forcing the model to rely more heavily on route planners and visual observations.

Datasets

The paper introduces a new dataset called Drive360.

  • Drive360: This dataset comprises 60 hours of driving data collected from multiple cities in Switzerland. It includes:
  • Videos from eight surround-view cameras.
  • Low-level driving maneuvers (steering angle and speed) recorded via the vehicle's CAN bus.
  • Two representations of planned routes (TomTom and OSM).
  • GPS-IMU data for vehicle odometry.

The dataset is notable for its high temporal resolution, 360-degree coverage, frame-wise synchronization, and diverse driving conditions.

Key Insights and Ideas

  • 360-degree vision is crucial for safety: Surround-view cameras significantly reduce failures in complex urban scenarios like city driving and intersections, where a single front-view camera is insufficient.
  • Route planners are powerful aids: Incorporating route planner information, especially visual representations like the TomTom map, substantially improves driving model performance, particularly for steering angle prediction.
  • Visual route representations outperform raw GPS: Rendered map sequences from navigation apps provide a more effective representation of planned routes than raw GPS coordinates for driving model training.
  • Relying less on past maneuvers can be beneficial: By removing historical driving maneuvers from the input, the model is forced to learn more from route planners and visual context, leading to better generalization.
  • Human driving is a complex baseline: Even with rich sensor data, predicting human-like maneuvers accurately remains challenging, especially in nuanced situations.

Key Results

  • Improved Steering Angle Prediction: The proposed model with surround-view cameras and the TomTom route planner achieved a Mean Squared Error (MSE) of 0.134 for steering angle prediction on the full dataset, outperforming single-camera methods (e.g., 0.869 for front-view only) and other end-to-end approaches.
  • Improved Speed Prediction: Similarly, speed prediction also saw significant gains, with an MSE of 0.0030 for the proposed method compared to 0.0147 for a front-view only baseline.
  • Intersection Performance: In intersection scenarios, surround-view cameras significantly improved speed prediction accuracy (e.g., MSE of 0.080 for surround-view vs. 0.118 for front-view at <10 km/h).
  • Qualitative Improvements: Visualizations demonstrated that the surround-view + route planner model could better anticipate maneuvers like right and left turns and navigate roundabouts with pedestrians more effectively than a front-camera-only model.

Limitations and Future Work

  • Synchronization Error: While efforts were made, there's a slight synchronization error between sensor streams (up to 8.3 ms for video, 10 ms for CAN bus), which could be improved with hardware triggers.
  • OSM Route Planner Performance: The numerical OSM route planner did not yield as significant improvements as the visual TomTom planner, possibly due to differences in map quality and representation.
  • Future Work: The authors suggest exploring more sophisticated ways to balance the steering and speed prediction losses, further investigating the benefits of different route planner representations, and evaluating the model in more diverse and challenging driving situations. They also mention that integrating traffic agent detection and tracking is a future research direction.

Summary updated

Abstract

For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: 1) by representing the planned routes on OpenStreetMap as a stack of GPS coordinates, and 2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: 1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and 2) route planners help the driving task significantly, especially for steering angle prediction.
BibTeX
@inproceedings{hecker2018end,
  author = {Simon Hecker and Dengxin Dai and Luc Van Gool},
  title = {{End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01234-2\_27},
}
Figure