Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation

Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, Daniela RusIEEE RA-L 2020 · OpenReview

Abstract

In this work, we present a data-driven simulation and training engine capable of learning end-to-end autonomous vehicle control policies using only sparse rewards. By leveraging real, human-collected trajectories through an environment, we render novel training data that allows virtual agents to drive along a continuum of new local trajectories consistent with the road appearance and semantics, each with a different view of the scene. We demonstrate the ability of policies learned within our simulator to generalize to and navigate in previously unseen real-world roads, without access to any human control labels during training. Our results validate the learned policy onboard a full-scale autonomous vehicle, including in previously un-encountered scenarios, such as new roads and novel, complex, near-crash situations. Our methods are scalable, leverage reinforcement learning, and apply broadly to situations requiring effective perception and robust operation in the physical world.

Introduction: The Challenge of End-to-End Driving

End-to-end autonomous driving—where a neural network maps raw pixels directly to steering commands—typically relies on Imitation Learning (IL). While IL is data-efficient, it is notoriously fragile. Because human drivers rarely veer off the road, training data lacks "recovery" examples. When an IL model makes a small mistake, it enters an unfamiliar state (off-center or off-orientation) and quickly cascades into a crash.

Traditional simulators (like CARLA) offer a safe environment to learn these recoveries, but they suffer from the "reality gap": synthetic textures and simplified physics often fail to prepare a model for the visual complexity of the real world, such as shadows, road cracks, and varying vegetation. Collecting real-world recovery data is both dangerous and prohibitively expensive. To address this, researchers from MIT CSAIL developed VISTA (Virtual Image Synthesis and Transformation for Autonomy).

Overview of the VISTA pipeline, illustrating the transition from data-driven simulation to real-world deployment.

VISTA: Bridging the Reality Gap

VISTA bridges the gap between real-world data and simulation by synthesizing photorealistic new driving trajectories from a small amount of real-world human driving data. Instead of building a 3D world from scratch, VISTA takes a single monocular video stream and odometry from a human driver and uses a CNN to estimate a depth map for each frame.

To simulate an agent being "off-course," VISTA projects the pixels into 3D space using the depth map, applies a relative transformation (rotation or translation) based on the difference between agent and human dynamics, and re-projects them back into a 2D camera view. This allows a virtual agent to "drive" along an infinite continuum of new local trajectories that are photorealistic and semantically consistent with the original scene.

Technical architecture of VISTA, showing the 3D view synthesis process and the geometry of relative transformations between trajectories.

Learning via Reinforcement

Unlike IL, which requires human labels, the agent in VISTA is trained via Reinforcement Learning (RL) using Policy Gradient methods. It receives a simple sparse reward: +1 for every meter driven without exiting the lane boundaries. The agent learns to stay centered not by copying a human, but by exploring the synthesized space and learning that lane stability is the optimal strategy for maximizing reward. This approach allows the agent to experience "near-crash" scenarios—such as being 1.5 meters off-center or rotated 30 degrees—that would be impossible to collect safely in reality.

Key Results and Real-World Validation

The effectiveness of VISTA was validated through zero-shot transfer to a full-scale 2015 Toyota Prius V testbed on a challenging 3km rural road environment. Key findings include:

  • Zero-Shot Transfer: The VISTA-trained agent achieved 0.0 interventions over 10km of autonomous driving in the real world without any fine-tuning.
  • Superior Robustness: In "near-crash" recovery tests, VISTA-trained agents recovered successfully twice as frequently as the best imitation learning models.
  • Outperforming Sim-to-Real: Models trained in CARLA (even with advanced domain adaptation) struggled significantly in the real world compared to VISTA, highlighting the importance of photorealism in closing the reality gap.

Discussion and Future Work

While VISTA currently focuses on static environments and lateral control (steering), it demonstrates that re-rendering real pixels preserves the high-level semantic complexities crucial for generalization. Future work aims to incorporate dynamic obstacles like other cars and pedestrians, as well as extending the framework to complex maneuvers like intersections and point-to-point navigation.

Summary updated

Abstract

In this work, we present a data-driven simulation and training engine capable of learning end-to-end autonomous vehicle control policies using only sparse rewards. By leveraging real, human-collected trajectories through an environment, we render novel training data that allows virtual agents to drive along a continuum of new local trajectories consistent with the road appearance and semantics, each with a different view of the scene. We demonstrate the ability of policies learned within our simulator to generalize to and navigate in previously unseen real-world roads, without access to any human control labels during training. Our results validate the learned policy onboard a full-scale autonomous vehicle, including in previously un-encountered scenarios, such as new roads and novel, complex, near-crash situations. Our methods are scalable, leverage reinforcement learning, and apply broadly to situations requiring effective perception and robust operation in the physical world.
BibTeX
@article{amini2020learning,
  author = {Alexander Amini and Igor Gilitschenski and Jacob Phillips and Julia Moseyko and Rohan Banerjee and Sertac Karaman and Daniela Rus},
  title = {{Learning Robust Control Policies for End-to-End Autonomous Driving From Data-Driven Simulation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=fUY5Y-TfX46},
}
Figure