Training autonomous vehicles to navigate complex traffic is notoriously difficult. While reinforcement learning (RL) agents can learn through trial and error in simulators, doing so in the real world is dangerous and expensive. Conversely, we have massive amounts of observational traffic data, but training policies purely offline from this data often leads to catastrophic failures when deployed.
Presented at ICLR 2019, this research tackles the "holy grail" of autonomous driving: learning robust driving policies in dense traffic using only recorded observations, without any real-world trial and error or simulator interaction during training. The authors introduce Model-Predictive Policy Learning with Uncertainty Regularization (MPUR) to bridge the gap between offline data and reliable policy execution.
The Challenge of Offline Reinforcement Learning
When training an agent purely on observational data (like traffic camera footage), the primary hurdle is distribution shift.
If a policy is trained on a fixed dataset, it will eventually attempt an action that leads to a state slightly outside the training distribution. Because the learned dynamics model has never seen this state, it may make wildly incorrect—and often overly optimistic—predictions. For example, the model might hallucinate that driving off the road at 80 mph incurs zero cost. The policy then "exploits" this hallucination, leading to catastrophic failure when deployed in a real or simulated environment.
Methodology: MPUR
To prevent the policy from exploiting these model errors, the researchers propose a two-stage approach that explicitly penalizes the agent for venturing into unknown territory.
1. Action-Conditional Forward Model
First, they build a stochastic dynamics model based on Variational Autoencoders (VAEs). This model predicts the next state given the current state and a chosen action. To handle the inherent randomness of real-world traffic (aleatoric uncertainty), the model incorporates latent variables.
To prevent a common failure mode where stochastic models ignore the action input and rely entirely on the latent variable to explain state transitions, the authors introduce a z-dropout trick. By occasionally dropping out the latent information during training, the model is forced to remain sensitive to the agent's actions.
2. Uncertainty Regularization (The Core Innovation)
The "secret sauce" of MPUR is adding an uncertainty cost ($U$) to the standard driving objective (e.g., staying in lanes, avoiding collisions). This cost acts as a differentiable guardrail, effectively telling the policy: "Optimize for speed and safety, but do not enter states I cannot reliably predict."
They measure this uncertainty using Monte Carlo Dropout. By running the same input through the forward model multiple times with different dropout masks, they generate a distribution of predictions.
- Low Variance: The model has seen similar data and is confident.
- High Variance: The model is guessing.
The policy is penalized for moving toward regions with high variance. Crucially, this approach elegantly separates aleatoric uncertainty (the world is inherently random) from epistemic uncertainty (the model is ignorant). By only penalizing epistemic uncertainty, the agent can handle complex, multi-modal traffic without being paralyzed by the natural unpredictability of other drivers.
Because both the dynamics model and the uncertainty estimator are differentiable, the policy can be trained via backpropagation through time (BPTT) over long "unrolled" futures (up to 40 time steps).
Experimental Setup and Key Results
The authors trained their model using the NGSIM I-80 dataset, which contains 45 minutes of real-world highway traffic footage recorded from overhead cameras (yielding roughly 2 million state-action transitions). The state representation consisted of a 3-channel top-down image (lanes, other cars, and the ego car) alongside a vector for position and velocity. They evaluated the policy in a "cold-start" environment where the ego car is controlled by the learned policy, while all other cars follow their historically recorded trajectories.
MPUR was compared against several baselines, including standard Imitation Learning and unregularized model-based RL:
- High Success Rate: MPUR achieved a 74.8% success rate (successfully navigating to the end of the road without a collision).
- Failure of Baselines: Standard model-based methods lacking uncertainty regularization had a 0% success rate. They immediately exploited model inaccuracies, imagining they could "teleport" or drive through other cars.
- Sample Efficiency: Compared to Proximal Policy Optimization (PPO)—a state-of-the-art model-free method—MPUR reached high performance without a single "real" environment interaction. PPO required thousands of simulated crashes to reach a comparable level.
Limitations and Future Directions
While MPUR demonstrates a massive leap for offline policy learning, limitations remain. The system still falls short of human performance, which achieves a 100% success rate in these specific scenarios. Additionally, the evaluation environment is non-reactive; other cars follow fixed historical paths and do not react to the ego car's novel behaviors (e.g., braking if cut off). Finally, calculating uncertainty via multiple dropout passes at every training step is computationally expensive.
Future work will likely explore more sophisticated, computationally efficient uncertainty estimators and test the approach in more complex urban environments featuring intersections and pedestrians.
Summary updated