Self Training Autonomous Driving Agent

Shashank Kotyan, Danilo Vasconcellos Vargas, Venkanna Udutalapally

Abstract

This paper presents the Self Training Autonomous Driving (STAD) agent, an optimized architecture for autonomous driving based on the World Models framework. By incorporating "difference images"—the subtraction of consecutive video frames—into the encoding process, the model explicitly captures motion and environmental changes. Evaluated on the OpenAI CarRacing-v0 environment, STAD achieves state-of-the-art performance with significantly higher sample efficiency, requiring 96% fewer agents and 90% fewer rollouts than the original World Models implementation. Our results demonstrate that prioritizing motion features leads to faster convergence and greater stability in reinforcement learning for navigation tasks.

1. Introduction

Training autonomous vehicles via Reinforcement Learning (RL) is often hindered by extreme sample inefficiency, requiring millions of simulated miles to learn basic maneuvers. While architectures like World Models have improved efficiency by building internal representations of the environment, they remain computationally expensive. This research addresses these challenges by introducing "difference images" to focus the agent's attention on motion and environmental dynamics, thereby accelerating the learning process.

2. Methodology

The STAD architecture builds upon the World Models framework, which decomposes the driving task into three primary components: visual perception, temporal prediction, and motor control.

The STAD pipeline, integrating an Encoder-Decoder, MDN-RNN, and Controller within the simulation environment.

2.1 Difference Autoencoder

In standard World Models, a Variational Autoencoder (VAE) compresses raw camera frames into a latent vector. STAD modifies this by utilizing a Difference Autoencoder. This encoder takes both the current frame and a difference image (the subtraction of the previous frame from the current one) as input. This approach forces the latent space to represent motion more aggressively, providing the network with a direct signal for physics and trajectory.

2.2 MDN-RNN and Controller

The latent variables are processed by a Recurrent Neural Network with a Mixture Density Network (MDN-RNN), which predicts future latent states. Finally, a simple Controller network decides on steering, acceleration, and braking based on the RNN's output. The controller is optimized using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES).

2.3 Training Environment

The model was tested in the OpenAI Gym CarRacing-v0 environment, a 2D top-down simulation with randomly generated tracks. To train the autoencoder, a small dataset of 1,000 rollouts was collected using random actions.

3. Results and Discussion

The STAD agent successfully solved the environment, reaching a score of 900+ with staggering efficiency gains. By explicitly feeding motion signals into the encoder, the model localized the "region of interest," allowing the agent to focus on the car's trajectory rather than static background elements.

Average episodic reward over 600 generations. Models utilizing difference images (blue, orange, red) significantly outperform the baseline (green) in both speed of convergence and final reward.

Key performance metrics include:

  • 96% reduction in total agents evaluated (4,800 vs. 128,000 in the original study).
  • 90% reduction in required training rollouts (1,000 vs. 10,000).
  • 70% faster convergence, reaching peak performance in 600 generations compared to 2,000.

As shown in Figure 6, the inclusion of difference images (DiffOutput, DiffInputOutput) leads to much faster reward growth and earlier stability compared to the baseline model (No DiffInputOutput).

4. Conclusion

STAD demonstrates that simple modifications to input features, such as the inclusion of difference images, can yield massive improvements in RL sample efficiency. While the current model is optimized for 2D top-down views, future work will focus on developing view-invariant models capable of handling the complex optical flow of first-person 3D environments like CARLA.

Summary updated

BibTeX
@inproceedings{kotyan2019self,
  author = {Shashank Kotyan and Danilo Vasconcellos Vargas and Venkanna Udutalapally},
  title = {{Self Training Autonomous Driving Agent}},
  booktitle = {Sice},
  year = {2019},
  doi = {10.23919/sice.2019.8859883},
}
Figure