Dynamic Input for Deep Reinforcement Learning in Autonomous Driving

Maria Hügle, Gabriel Kalweit, Branka Mirchevska, Moritz Werling, Joschka Boedecker

This paper, presented by researchers from the University of Freiburg and the BMW Group, tackles a fundamental bottleneck in applying Deep Reinforcement Learning (DRL) to autonomous driving: the "fixed-input" problem.

TL;DR

The researchers introduce DeepSet-Q, a reinforcement learning architecture that uses "Deep Sets" to process a variable number of surrounding vehicles. Unlike traditional models that rely on fixed-size grids or limited neighbor lists, DeepSet-Q is permutation-invariant and generalizes significantly better to complex, unseen traffic scenarios.

What Problem Does This Paper Address?

In autonomous driving, the environment is highly dynamic. At one moment, you might be alone on a highway; the next, you are surrounded by a dozen cars.

Most DRL agents are built with fixed-size input layers. To make this work, engineers usually employ one of two "hacks":

  1. The $n$-nearest approach: Only consider the $n$ closest cars. If the $(n+1)$-th car is the one about to cause a collision, the agent is effectively blind to it.
  2. Occupancy Grids: Represent the world as a top-down pixelated grid. This is computationally expensive, and if a car is outside the predefined grid boundaries, it doesn't exist to the agent.

Furthermore, standard neural networks are sensitive to the order of inputs. If you swap the positions of "Car A" and "Car B" in a list, a standard network might output a different action, even though the physical reality hasn't changed. This paper seeks a method that is both flexible in size and "permutation invariant."

Methods

The authors propose DeepSet-Q, integrating the "Deep Sets" architecture into a Deep Q-Network (DQN) framework.

  • Deep Sets Architecture: Instead of a single large vector, the input is a set of feature vectors (one for each detected car).
  • Each car's data is processed by the same small network ($\phi$).
  • The outputs are then combined using a symmetric pooling operation (like a sum or max). This ensures that the order of cars in the list doesn't matter.
  • A second network ($\rho$) processes this combined "set representation" to help decide the best driving action.
  • Set2Set-Q: As a more complex baseline, they also implemented an architecture using an LSTM with an attention mechanism to process the set.
  • Baselines: They compared these against a Relational Grid (fixed list of neighbors), a CNN (occupancy grid), and a standard Rule-based lane-change controller.

Datasets

The study used the SUMO (Simulation of Urban MObility) traffic simulator.

  • Environment: A 1000m circular highway with 3 to 5 lanes.
  • Dataset 1: 500,000 samples featuring an arbitrary number of surrounding vehicles.
  • Dataset 2 (The Generalization Test): 500,000 samples containing at most six surrounding vehicles. This was used to see if an agent trained on simple traffic could handle dense, crowded highways.

Key Insights and Ideas

  • Structural Generalization: The "aha moment" is that Deep Sets don't just handle variable inputs; they understand the concept of a vehicle. Because the $\phi$ network is applied to every car identically, the model learns universal features of "car-ness" that apply whether there are 2 cars or 20.
  • The Grid Bottleneck: Occupancy grids (CNNs) struggle when the agent moves to a 5-lane highway if they were only trained on 3 lanes. Deep Sets don't care about the number of lanes; they only care about the objects within them.
  • Permutation Invariance is Key: By forcing the model to be invariant to the order of vehicles, the researchers reduced the "search space" the model had to learn, leading to faster convergence and more stable behavior.

Key Results

  • Top Performance: DeepSet-Q achieved the highest mean reward and the lowest variance across all tested traffic densities (30 to 90 vehicles in the scenario).
  • Superior Generalization: When trained on Dataset 2 (max 6 cars) and tested on dense traffic, DeepSet-Q maintained high performance. In contrast, the CNN and Fixed-Input models saw their performance plummet because they couldn't process the "extra" vehicles they hadn't seen during training.
  • Robustness: The authors added Gaussian noise to the distance and velocity sensors. DeepSet-Q remained robust, showing only a minor decrease in performance even with significant sensor noise.

Limitations and Future Work

  • Object Diversity: The current model focuses primarily on other passenger cars. The authors note that future iterations should include heterogeneous sets containing pedestrians, cyclists, and traffic lights.
  • Temporal Context: While the model handles the spatial set well, it doesn't explicitly model the history of those objects (e.g., is that car accelerating or braking?). Incorporating recurrent units (RNNs) into the $\phi$ network could address this.
  • Sim-to-Real: The work is entirely simulation-based. Moving from the "perfect" object lists of SUMO to the noisy, occluded detections of real-world LiDAR and cameras remains a significant hurdle.

Summary updated

Abstract

In many real-world decision making problems, reaching an optimal decision requires taking into account a variable number of objects around the agent. Autonomous driving is a domain in which this is especially relevant, since the number of cars surrounding the agent varies considerably over time and affects the optimal action to be taken. Classical methods that process object lists can deal with this requirement. However, to take advantage of recent high-performing methods based on deep reinforcement learning in modular pipelines, special architectures are necessary. For these, a number of options exist, but a thorough comparison of the different possibilities is missing. In this paper, we elaborate limitations of fully-connected neural networks and other established approaches like convolutional and recurrent neural networks in the context of reinforcement learning problems that have to deal with variable sized inputs. We employ the structure of Deep Sets in off-policy reinforcement learning for high-level decision making, highlight their capabilities to alleviate these limitations, and show that Deep Sets not only yield the best overall performance but also offer better generalization to unseen situations than the other approaches.
BibTeX
@inproceedings{hugle2019dynamic,
  author = {Maria Hügle and Gabriel Kalweit and Branka Mirchevska and Moritz Werling and Joschka Boedecker},
  title = {{Dynamic Input for Deep Reinforcement Learning in Autonomous Driving}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=3PZ9aX6plGw},
}

Loading related content...

Figure