Abstract
Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.
Introduction: The Challenge of Covariate Shift
Learning to drive in dense traffic typically necessitates either extensive, potentially hazardous real-world interaction or high-fidelity, costly simulations. Offline learning from human driving datasets offers a safer alternative but is plagued by covariate shift. When a learned policy deviates even slightly from the training distribution, it enters states where the dynamics model's predictions become unreliable.
In these "off-track" states, models often become "wrongly optimistic," predicting safe outcomes where collisions should occur. Policy optimizers then exploit these hallucinations, selecting actions that appear successful in "imagination" but fail in reality.
Methodology: MPUR
We introduce Model-Predictive Policy with Uncertainty Regularization (MPUR). Our approach involves two primary components:
1. Action-Conditional Forward Model
We train a stochastic dynamics model based on Variational Autoencoders (VAEs) to predict future states given past states and actions. To ensure the model remains sensitive to the policy's actions—rather than "explaining away" the future via the latent variable $z$—we implement z-dropout, a mixture-of-Gaussians posterior.
2. Uncertainty Regularization
The policy network is trained by unrolling trajectories through the forward model. The total loss function incorporates a standard Policy Cost (e.g., lane keeping, collision avoidance) and a novel Uncertainty Cost.
This uncertainty is estimated using Monte Carlo Dropout. By passing the same input through the model multiple times with different dropout masks, we calculate the variance of the predictions. High variance indicates model uncertainty. By penalizing the policy for entering high-uncertainty regions, we force it to stay within the manifold of the training data. Crucially, this uncertainty estimate is differentiable, allowing for direct gradient-based optimization of the policy.
Experimental Setup
We evaluated MPUR using the NGSIM I-80 dataset, which consists of approximately 2 million transitions from real-world highway traffic in California. We transformed this data into a "Gym-like" environment where an "ego-vehicle" is controlled by the agent while other vehicles follow their recorded trajectories.
Results and Insights
MPUR significantly outperformed baseline methods:
- Success Rate: MPUR achieved a 74.8% success rate in reaching the destination without collisions.
- The Regularization Gap: Standard Value Gradient (VG) methods without uncertainty regularization failed completely (0% success rate), as they learned to exploit model errors.
- Planning Horizon: Simple 1-step Imitation Learning (IL) only reached a 7.3% success rate, highlighting the necessity of long-horizon planning for complex navigation.
Key insights include the role of differentiable uncertainty as a "guardrail" and the importance of action sensitivity in stochastic models. Deterministic models were found to be inadequate as they lack uncertainty estimates and tend to produce blurry, averaged predictions.
Conclusion and Future Work
MPUR demonstrates that effective driving policies can be learned from purely observational data by explicitly accounting for model uncertainty. Current limitations include the non-reactive nature of other agents in the test environment and a performance gap compared to human drivers. Future work will explore optimizing for more complex maneuvers and improving the realism of the evaluation environment.
Summary updated