Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Mikael Henaff, Alfredo Canziani, Yann LeCun

Abstract

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Introduction: The Challenge of Covariate Shift

Learning to drive in dense traffic typically necessitates either extensive, potentially hazardous real-world interaction or high-fidelity, costly simulations. Offline learning from human driving datasets offers a safer alternative but is plagued by covariate shift. When a learned policy deviates even slightly from the training distribution, it enters states where the dynamics model's predictions become unreliable.

Learned functions can hallucinate low-cost regions in areas with no training data, leading to policy exploitation.

In these "off-track" states, models often become "wrongly optimistic," predicting safe outcomes where collisions should occur. Policy optimizers then exploit these hallucinations, selecting actions that appear successful in "imagination" but fail in reality.

Methodology: MPUR

We introduce Model-Predictive Policy with Uncertainty Regularization (MPUR). Our approach involves two primary components:

1. Action-Conditional Forward Model

We train a stochastic dynamics model based on Variational Autoencoders (VAEs) to predict future states given past states and actions. To ensure the model remains sensitive to the policy's actions—rather than "explaining away" the future via the latent variable $z$—we implement z-dropout, a mixture-of-Gaussians posterior.

2. Uncertainty Regularization

The policy network is trained by unrolling trajectories through the forward model. The total loss function incorporates a standard Policy Cost (e.g., lane keeping, collision avoidance) and a novel Uncertainty Cost.

The MPUR architecture uses dropout masks to estimate model uncertainty, which is then used as a regularization term in the policy cost.

This uncertainty is estimated using Monte Carlo Dropout. By passing the same input through the model multiple times with different dropout masks, we calculate the variance of the predictions. High variance indicates model uncertainty. By penalizing the policy for entering high-uncertainty regions, we force it to stay within the manifold of the training data. Crucially, this uncertainty estimate is differentiable, allowing for direct gradient-based optimization of the policy.

Experimental Setup

We evaluated MPUR using the NGSIM I-80 dataset, which consists of approximately 2 million transitions from real-world highway traffic in California. We transformed this data into a "Gym-like" environment where an "ego-vehicle" is controlled by the agent while other vehicles follow their recorded trajectories.

Results and Insights

MPUR significantly outperformed baseline methods:

  • Success Rate: MPUR achieved a 74.8% success rate in reaching the destination without collisions.
  • The Regularization Gap: Standard Value Gradient (VG) methods without uncertainty regularization failed completely (0% success rate), as they learned to exploit model errors.
  • Planning Horizon: Simple 1-step Imitation Learning (IL) only reached a 7.3% success rate, highlighting the necessity of long-horizon planning for complex navigation.

Key insights include the role of differentiable uncertainty as a "guardrail" and the importance of action sensitivity in stochastic models. Deterministic models were found to be inadequate as they lack uncertainty estimates and tend to produce blurry, averaged predictions.

Conclusion and Future Work

MPUR demonstrates that effective driving policies can be learned from purely observational data by explicitly accounting for model uncertainty. Current limitations include the non-reactive nature of other agents in the test environment and a performance gap compared to human drivers. Future work will explore optimizing for more complex maneuvers and improving the realism of the evaluation environment.

Summary updated

Abstract

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.
BibTeX
@inproceedings{henaff2019model,
  author = {Mikael Henaff and Alfredo Canziani and Yann LeCun},
  title = {{Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=u5sZi0ms3v},
}
Figure