TL;DR
This paper tackles the challenge of building more robust end-to-end driving models by moving beyond single front-facing cameras. They introduce a new dataset and a deep learning architecture that integrates surround-view camera data and route planner information to predict steering angle and speed, showing significant improvements over single-camera approaches, especially in complex urban scenarios.
What Problem Does This Paper Address?
Current end-to-end driving models often rely on a single front-facing camera, which provides a limited view of the environment. This limitation makes it difficult for models to learn safe and effective driving behaviors, particularly in scenarios requiring awareness of surroundings (e.g., intersections, lane changes) or when navigating complex routes. Human drivers, in contrast, utilize a 360-degree view (mirrors) and an internal "mental map" for navigation. The paper aims to bridge this gap by developing driving models that incorporate richer sensory input, mimicking human drivers' use of comprehensive visual information and route planning.
Methods
The core of the paper is a novel deep learning architecture designed for end-to-end driving. This architecture takes as input:
- Surround-view camera data: Videos from eight cameras providing a 360-degree view of the vehicle's surroundings. These are processed by ResNet34 (pre-trained on ImageNet) for feature extraction.
- Route planner information: Two representations are used:
- TomTom Map: Rendered video sequences from the TomTom Go Mobile app.
- OpenStreetMap (OSM): A sequence of GPS coordinates representing the planned route.
- Historical vehicle states: Previous steering angles and speeds.
These features are fused and fed into a multi-layer LSTM network to capture temporal dependencies. The fused information is then passed through fully-connected layers to predict the future steering angle and vehicle speed. The authors highlight that they also experimented with a setting that excludes historical driving maneuvers, forcing the model to rely more heavily on route planners and visual observations.
Datasets
The paper introduces a new dataset called Drive360.
- Drive360: This dataset comprises 60 hours of driving data collected from multiple cities in Switzerland. It includes:
- Videos from eight surround-view cameras.
- Low-level driving maneuvers (steering angle and speed) recorded via the vehicle's CAN bus.
- Two representations of planned routes (TomTom and OSM).
- GPS-IMU data for vehicle odometry.
The dataset is notable for its high temporal resolution, 360-degree coverage, frame-wise synchronization, and diverse driving conditions.
Key Insights and Ideas
- 360-degree vision is crucial for safety: Surround-view cameras significantly reduce failures in complex urban scenarios like city driving and intersections, where a single front-view camera is insufficient.
- Route planners are powerful aids: Incorporating route planner information, especially visual representations like the TomTom map, substantially improves driving model performance, particularly for steering angle prediction.
- Visual route representations outperform raw GPS: Rendered map sequences from navigation apps provide a more effective representation of planned routes than raw GPS coordinates for driving model training.
- Relying less on past maneuvers can be beneficial: By removing historical driving maneuvers from the input, the model is forced to learn more from route planners and visual context, leading to better generalization.
- Human driving is a complex baseline: Even with rich sensor data, predicting human-like maneuvers accurately remains challenging, especially in nuanced situations.
Key Results
- Improved Steering Angle Prediction: The proposed model with surround-view cameras and the TomTom route planner achieved a Mean Squared Error (MSE) of 0.134 for steering angle prediction on the full dataset, outperforming single-camera methods (e.g., 0.869 for front-view only) and other end-to-end approaches.
- Improved Speed Prediction: Similarly, speed prediction also saw significant gains, with an MSE of 0.0030 for the proposed method compared to 0.0147 for a front-view only baseline.
- Intersection Performance: In intersection scenarios, surround-view cameras significantly improved speed prediction accuracy (e.g., MSE of 0.080 for surround-view vs. 0.118 for front-view at <10 km/h).
- Qualitative Improvements: Visualizations demonstrated that the surround-view + route planner model could better anticipate maneuvers like right and left turns and navigate roundabouts with pedestrians more effectively than a front-camera-only model.
Limitations and Future Work
- Synchronization Error: While efforts were made, there's a slight synchronization error between sensor streams (up to 8.3 ms for video, 10 ms for CAN bus), which could be improved with hardware triggers.
- OSM Route Planner Performance: The numerical OSM route planner did not yield as significant improvements as the visual TomTom planner, possibly due to differences in map quality and representation.
- Future Work: The authors suggest exploring more sophisticated ways to balance the steering and speed prediction losses, further investigating the benefits of different route planner representations, and evaluating the model in more diverse and challenging driving situations. They also mention that integrating traffic agent detection and tracking is a future research direction.
Summary updated