Learning to Drive (L2D) as a Low-Cost Benchmark for Real-World Reinforcement Learning

Ari Viitala, Rinu Boney, Yi Zhao, Alexander Ilin, Juho KannalaICAR 2021 · DOI

Introduction

Deep Reinforcement Learning (RL) has achieved remarkable success in simulated environments, yet transitioning these capabilities to the physical world remains a significant challenge. This "sim-to-real" gap arises because agents trained in idealized simulators often fail when faced with real-world noise, latency, and complex physics. While training directly in the real world avoids this gap, it is typically prohibitively expensive and difficult to reproduce.

The "Learning to Drive" (L2D) benchmark addresses these issues by providing a standardized, low-cost ($250) platform for training RL agents directly in the real world. Utilizing miniature "Donkey cars," the authors demonstrate that state-of-the-art RL algorithms can learn to navigate tracks from scratch in under five minutes of interaction, eventually outperforming both imitation learning baselines and human operators.

Hardware and Environment

The L2D platform is built upon the open-source Donkey car RC kit. To ensure its suitability as a robust RL benchmark, the authors integrated an Intel RealSense T265 camera for precise speed estimation and implemented a PD controller to manage throttle sensitivity.

The L2D physical setup featuring a Donkey car on a custom-built track.

The benchmark includes three distinct physical tracks with detailed dimensions to facilitate reproducibility. These tracks range from standard loops to more challenging layouts with sharp right-angle turns. The environment is characterized by high-dimensional visual observations, stochastic delays, and non-stationary dynamics, such as battery voltage drops and varying surface friction.

Methodology

The study evaluates three primary learning approaches:

  1. Imitation Learning (IL): A baseline where a neural network mimics a human driver using a dataset of 10,000 steps.
  2. Model-free RL (SAC + VAE): This approach utilizes Soft Actor-Critic (SAC) combined with a Variational Autoencoder (VAE). The VAE compresses 40x40 pixel camera frames into a 20-dimensional latent space, which serves as the input for the SAC agent.
  3. Model-based RL (Dreamer): This represents a pioneering real-world application of the Dreamer algorithm. Dreamer learns a "world model" of the environment and optimizes its policy through "latent imagination"—simulating potential future outcomes within its learned model to improve sample efficiency.

The reward signal is designed for simplicity: agents receive a penalty (-10) for disengagements (driving off-track) and a small positive reward (+1) for staying on the track.

Experimental Results

The L2D benchmark reveals several key insights into real-world RL. Most notably, agents using the Dreamer algorithm were able to complete laps from scratch in less than five minutes of real-world interaction.

Training trajectories and disengagements (marked with 'x') across the three benchmark tracks.

Performance and Efficiency

Dreamer proved to be the most sample-efficient algorithm, requiring roughly half the disengagements of the SAC+VAE approach to achieve robust performance across all tracks. In high-speed tests on Track 1, the SAC+VAE agent achieved the fastest lap time (4.92s), surpassing both the human operator (5.82s) and the imitation learning baseline (6.62s).

Generalization

The study also explored cross-track generalization. Agents trained on the standard layout of Track 1 generalized well to other tracks. However, agents trained exclusively on Track 3—which consists solely of sharp right-angle turns—struggled to adapt to the smoother curves found on other layouts.

Discussion and Future Work

The results suggest that real-world RL is more feasible than previously thought, often outperforming simulators that are "too easy" due to their lack of real-world complexities. However, the current L2D setup has limitations: the environment is static, and off-track detection relies on simple color-thresholding or manual intervention.

Future work aims to expand the L2D benchmark to include more complex outdoor environments with varying lighting, multi-agent scenarios involving multiple cars, and goal-directed navigation tasks beyond simple loop-following.

Summary updated

BibTeX
@inproceedings{viitala2021learning,
  author = {Ari Viitala and Rinu Boney and Yi Zhao and Alexander Ilin and Juho Kannala},
  title = {{Learning to Drive (L2D) as a Low-Cost Benchmark for Real-World Reinforcement Learning}},
  booktitle = {ICAR},
  year = {2021},
  doi = {10.1109/icar53236.2021.9659342},
}
Figure