An LSTM-Based Autonomous Driving Model Using Waymo Open Dataset

Zhicheng Li, Zhihao Gu, Xuan Di, Rongye Shi

TL;DR

This paper tackles the challenge of understanding and replicating the driving policies of advanced autonomous vehicles (AVs) like Waymo, which are proprietary and inaccessible. They propose an LSTM-based model trained on the Waymo Open Dataset to predict future vehicle acceleration, demonstrating that incorporating camera imagery significantly improves prediction accuracy over using only numerical sensor data.

What Problem Does This Paper Address?

Developing robust autonomous driving systems requires understanding how AVs make decisions, particularly their driving policies. However, the exact programming and decision-making logic of commercial AVs, like those from Waymo, are kept secret by manufacturers. This lack of transparency forces academic researchers to make assumptions when building their own AV models or simulations, which may not accurately reflect real-world driving behavior and interactions. The paper aims to bridge this gap by learning an "imitation" driving policy directly from Waymo's data, allowing for more realistic AV research.

Methods

The core of the paper's approach is a Long Short-Term Memory (LSTM)-based model, chosen for its ability to process sequential data, which is crucial for understanding driving behavior over time. They explore two main model architectures:

  • Basic LSTM Model: This model takes 12 numerical features (kinematics and distances) derived from Waymo's sensor data as input. It uses an "encoder-decoder" structure, where the encoder processes the input sequence to create a latent representation, and the decoder uses this representation to predict future acceleration.
  • Advanced LSTM Model with Image Inputs: Recognizing the limitations of purely numerical data, this model incorporates camera images. They use a pre-trained Convolutional Neural Network (CNN) (specifically ResNetV2) as a feature extractor to convert images into meaningful vector representations. These image features are then combined with the 12 numerical features and fed into a similar encoder-decoder LSTM architecture. They experiment with using just the front camera and all five available cameras.

Datasets

The primary dataset used is the Waymo Open Dataset. This dataset is described as large, rich, and diverse, containing data from Waymo's level-5 autonomous vehicles across various traffic conditions. It includes sensor data (radar, lidar, cameras) and labels for objects, providing kinematics and spatial parameters. For their experiments, they used 32 training tars and 8 validation tars, extracting approximately 45,000 video segments.

Key Insights and Ideas

  • The "Black Box" Problem: The proprietary nature of AV driving policies is a significant barrier to academic research, necessitating methods to learn these policies indirectly.
  • LSTM for Driving Policy: LSTMs are well-suited for learning temporal dependencies in driving behavior, making them a strong candidate for predicting future actions like acceleration.
  • Image Data is Crucial: While numerical sensor data provides essential information, incorporating visual data from cameras dramatically improves the accuracy of predicting driving actions. This highlights that visual context is a key component of driving decisions.
  • Front Camera Sufficiency: Surprisingly, using only the front camera image yielded performance comparable to using all five cameras, suggesting that the front view captures the most critical information for predicting immediate future actions, while also being computationally more efficient.
  • Smoother Predictions: The proposed model, especially with image inputs and preprocessing, tends to produce smoother acceleration predictions compared to Waymo's actual driving behavior, which can be beneficial for passenger comfort.

Key Results

The LSTM-based models significantly outperformed baseline models (NN, CNN, XGBoost, etc.) in predicting acceleration.

  • The LSTM with 12 features achieved MAE X of 0.3179 and MAE Y of 0.3088.
  • The LSTM with front camera showed a substantial improvement, with MAE X of 0.1379 and MAE Y of 0.1278.
  • The LSTM with all cameras achieved the best results, with MAE X of 0.1327 and MAE Y of 0.1363, though the performance gain over the front-camera-only model was marginal.
  • Training time increased significantly with the addition of image data, with the LSTM with all cameras taking 37.8 hours to train, compared to 1.8 hours for the LSTM with 12 features.

Limitations and Future Work

  • "Online Approach" Limitation: The current model uses ground truth relative distances and velocities as input after predicting the first frame. This "online approach" can lead to accumulating errors in real-world scenarios where only predicted values are available.
  • Future Work:
    • Implement a "true online" prediction method where predictions from previous steps are used as input for subsequent steps to mitigate error accumulation.
  • Extend the model to generalize to other traffic scenarios beyond car-following, such as lane-changing, merging, and diverging.
  • Further explore the impact of incorporating more camera views and potentially other sensor modalities.

Summary updated

Abstract

The Waymo Open Dataset has been released recently, providing a platform to crowdsource some fundamental challenges for automated vehicles (AVs), such as 3D detection and tracking. While~the dataset provides a large amount of high-quality and multi-source driving information, people in academia are more interested in the underlying driving policy programmed in Waymo self-driving cars, which is inaccessible due to AV manufacturers' proprietary protection. Accordingly, academic researchers have to make various assumptions to implement AV components in their models or simulations, which may not represent the realistic interactions in real-world traffic. Thus, this paper introduces an approach to learn a long short-term memory (LSTM)-based model for imitating the behavior of Waymo's self-driving model. The proposed model has been evaluated based on Mean Absolute Error (MAE). The experimental results show that our model outperforms several baseline models in driving action prediction. In addition, a visualization tool is presented for verifying the performance of the model.
BibTeX
@inproceedings{li2020lstm,
  author = {Zhicheng Li and Zhihao Gu and Xuan Di and Rongye Shi},
  title = {{An LSTM-Based Autonomous Driving Model Using Waymo Open Dataset}},
  booktitle = {Applied Sciences},
  year = {2020},
  doi = {10.3390/app10062046},
}
Figure