Abstract
Deep learning has revolutionized the ability to learn “end-to-end” autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-to-point navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.
Introduction
Traditional "end-to-end" autonomous driving models map camera images directly to steering commands. However, these systems often struggle with multi-modality—for instance, at a fork in the road, a single-output model might average the two paths, leading to dangerous behavior. Furthermore, these models typically lack contextual localization, assuming perfect GPS and map data, which is rarely the case in real-world scenarios. This paper introduces a variational neural network that addresses these flaws by integrating raw sensory data with coarse-grained, noisy maps to enable both navigation and probabilistic localization.
Methodology
The core of the proposed system is a Variational End-to-End Network designed to handle uncertainty in both action and position.
Gaussian Mixture Model (GMM) Output
Instead of predicting a single steering angle, the network outputs parameters for a Gaussian Mixture Model (GMM), including weights ($\phi_i$), means ($\mu_i$), and variances ($\sigma_i$). This allows the model to represent multiple possible actions at ambiguous points, such as intersections.
Dual-Map Processing and Architecture
The architecture utilizes parallel Convolutional Neural Networks (CNNs) to process three camera views (front, left, and right) and a map patch. These features are merged into fully connected layers. The model can operate in two modes:
- Unrouted: Using a general road network map to understand all possible paths.
- Routed: Using a map with a specific path highlighted to produce a deterministic steering command for goal-oriented navigation.
Probabilistic Localization
The system employs a mathematical framework to update the vehicle's "belief" about its position. By matching visual features (e.g., the topology of an upcoming intersection) with the local map geometry, the model can refine its spatial and angular position estimates, effectively performing "rough localization" similar to a human driver.
Experimental Setup
The model was trained on 25 km of real-world driving data collected in suburban environments using a modified Toyota Prius V. Testing was conducted on entirely disjoint road segments and intersections. Map data was sourced from OpenStreetMap (OSM), providing coarse topological graphs. The system was implemented onboard using an NVIDIA Drive PX2 for real-time evaluation.
Key Insights
- Uncertainty as Information: A drop in the probability density of the steering command signals that the vehicle is approaching a decision point, such as an intersection.
- Visual-to-Map Correspondence: The network learns to associate visual "driving affordances" with geometric patterns in the map, enabling localization without precise GPS.
- Generalization: The model successfully navigated novel geometries, such as roundabouts, which were not present in the training set, indicating it learned general relationships rather than memorizing specific locations.
Results
The system demonstrated significant performance in both localization and real-time navigation:
- Localization Refinement: At intersections, spatial uncertainty was reduced by an average of 0.31 nats, and heading uncertainty by approximately 11 degrees (0.2 radians).
- Real-Time Performance: The full stack operates at 15Hz onboard a full-scale vehicle.
- Robustness: In "kidnapped robot" scenarios, the system correctly identified its true location among multiple candidates by matching visual topology to map geometry.
Conclusion
This work demonstrates that variational networks can effectively combine low-level sensory data with high-level, noisy map data for autonomous navigation. By explicitly modeling uncertainty, the system achieves robust performance in complex, multi-modal environments and provides a framework for human-like rough localization. Future work will focus on integrating these capabilities into continuous, closed-loop filters for online pose correction.
Summary updated