Learning Driving Models with a Surround-View Camera System and a Route Planner

Simon Hecker, Dengxin Dai, Luc Van Gool

Introduction

End-to-end autonomous driving models aim to map raw sensor inputs directly to vehicle control commands, such as steering and speed. However, most existing research is constrained by a reliance on single front-facing camera setups and a lack of navigational context. This limitation makes real-world driving tasks nearly intractable; for instance, without a route planner, a model cannot resolve the ambiguity of a fork in the road, leading to unsafe steering oscillations. Furthermore, human drivers rely on a 360-degree awareness to yield to cross-traffic and pedestrians—a capability missing in front-only models.

This paper addresses these gaps by introducing a model that utilizes a surround-view camera system and a route planner, providing the neural network with the same contextual tools available to a human driver. To support this research, the authors present the Drive360 dataset.

The Drive360 Dataset

The authors introduce Drive360, a novel dataset comprising 60 hours of driving data collected in and around Swiss cities. Unlike previous datasets that focus on semantic segmentation or limited camera views, Drive360 is explicitly designed for end-to-end learning with full spatial and navigational context.

Comparison of Drive360 with existing autonomous driving datasets.

Key features of the dataset include:

  • Surround-View Video: Synchronized footage from 8 GoPro cameras providing a full 360-degree field of view.
  • Vehicle State: High-frequency (50Hz) CAN bus data recording human steering and speed maneuvers.
  • Navigation Data: Synchronized route planner information provided in two formats: rendered visual maps from TomTom and raw GPS coordinates from OpenStreetMap (OSM).

Model Architecture

The proposed architecture frames driving as a continuous regression task, predicting the vehicle's steering angle and speed 0.3 seconds into the future based on a temporal history of inputs.

Surround-view and route planner fusion architecture.

Spatial and Temporal Processing

The model processes frames from four primary cameras (front, left, right, and rear). Each image is passed through a pre-trained ResNet34 CNN to extract spatial features. These features are then sequenced through Long Short-Term Memory (LSTM) networks to capture the temporal dynamics of the vehicle's motion.

Multi-Modal Fusion

Navigational context is integrated using either an AlexNet-based processor for rendered TomTom maps or a fully connected network for raw GPS coordinates. These features are concatenated with the temporal visual features in a final fusion layer. Notably, the model does not use its own past maneuvers as input, forcing it to learn control strictly from visual and navigational cues to avoid compounding feedback errors.

Key Insights and Results

The integration of surround-view and route planning yields significant improvements in driving performance, particularly in complex urban environments.

  • The "Fork in the Road" Solution: Providing a route planner significantly stabilizes steering. Adding the TomTom route planner to a front-view model reduced the Mean Squared Error (MSE) for steering from 0.967 to 0.808. In complex intersection maneuvers (<30 km/h), the Surround+TomTom model achieved a steering MSE of 3.214, vastly outperforming the front-view-only baseline (4.053).
  • Visual Maps vs. Raw GPS: Rendered visual maps outperformed raw GPS coordinates. The authors suggest that visual navigation apps provide superior "map-matching" and semantic context (e.g., distinguishing highway exits from slight bends) that raw coordinates lack.
  • Surround-View for Yielding: While front-facing cameras suffice for highway driving, surround-view is crucial for speed control in cities. At intersections (<10 km/h), the surround-view model achieved a speed MSE of 0.080, compared to 0.118 for the front-view model, as it could detect cross-traffic and pedestrians in blind spots.

Conclusion

By moving beyond the single-camera paradigm, this work demonstrates that surround-view vision and route planning are essential for realistic end-to-end driving. The Drive360 dataset provides a benchmark for future research in multi-modal autonomous control. Future work will explore more sophisticated loss function weighting and the integration of explicit object tracking to move toward a robust mediated perception approach.

Summary updated

BibTeX
@inproceedings{hecker2018learning,
  author = {Simon Hecker and Dengxin Dai and Luc Van Gool},
  title = {{Learning Driving Models with a Surround-View Camera System and a Route Planner}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01234-2\_27},
}
Figure