Dynamic Input for Deep Reinforcement Learning in Autonomous Driving

Maria Hügle, Gabriel Kalweit, Branka Mirchevska, Moritz Werling, Joschka Boedecker

Abstract

Autonomous driving requires agents to navigate environments with a highly variable number of actors. Traditional Deep Reinforcement Learning (DRL) models often rely on fixed-size inputs, such as occupancy grids or $n$-nearest neighbor lists, which limit flexibility and generalization. This paper introduces DeepSet-Q, an architecture that utilizes "Deep Sets" to process a variable number of surrounding vehicles. The approach is permutation-invariant and demonstrates significantly better performance and robustness in dense, unseen traffic scenarios compared to standard fixed-input baselines.

1. Introduction: The Fixed-Input Bottleneck

In autonomous driving, the environment is inherently dynamic. An agent may be alone on a highway one moment and surrounded by dozens of vehicles the next. Most DRL agents are constrained by fixed-size input layers, forcing researchers to adopt suboptimal "hacks." One common approach is the $n$-nearest neighbor method, which only considers a fixed number of nearby cars. This can leave the agent "blind" to critical vehicles just outside the selection. Another approach, Occupancy Grids, represents the world as a top-down pixelated grid. However, these are computationally expensive and limited by predefined boundaries.

Furthermore, standard neural networks are sensitive to the order of inputs. Swapping the positions of two vehicles in an input list can result in different actions, even if the physical state remains identical. This paper addresses these challenges by proposing a method that is both flexible in input size and permutation-invariant.

2. The DeepSet-Q Architecture

The researchers from the University of Freiburg and the BMW Group propose DeepSet-Q, which integrates the Deep Sets architecture into a Deep Q-Network (DQN) framework. Unlike traditional models that use a single large vector, DeepSet-Q treats the input as a set of feature vectors, one for each detected vehicle.

DeepSet-Q architecture: Individual vehicle features $x_t^i$ are processed by $\phi$, summed, and passed to $\rho$ to generate Q-values.

As shown in the architecture above, each vehicle's data is processed by the same small network ($\phi$). These outputs are then combined using a symmetric pooling operation (such as a sum), ensuring that the order of vehicles in the list does not affect the outcome. Finally, a second network ($\rho$) processes this combined representation to determine the optimal driving action. This structure allows the model to learn universal features of "car-ness" that apply regardless of the total number of vehicles present.

3. Experimental Methodology

The study utilized the SUMO (Simulation of Urban MObility) traffic simulator to create a 1000m circular highway environment with 3 to 5 lanes. The agents were tested on various scenarios, including standard highway driving and complex merging tasks.

Simulated highway scenarios used to evaluate agent performance in multi-lane and merging conditions.

To test generalization, the researchers trained the models on a dataset containing at most six surrounding vehicles and then evaluated them in dense traffic scenarios with up to 90 vehicles. This "Generalization Test" aimed to determine if an agent trained on simple traffic could handle crowded highways without additional training.

4. Key Results and Insights

DeepSet-Q achieved the highest mean reward and lowest variance across all tested traffic densities. Its primary advantage lies in structural generalization: because the $\phi$ network is applied identically to every car, the model understands the concept of a vehicle as an object rather than a fixed coordinate in a grid.

  • Superior Generalization: While CNN-based occupancy grids and fixed-input models saw performance plummet when faced with more vehicles than seen during training, DeepSet-Q maintained high performance.
  • Robustness to Noise: The model remained robust even when Gaussian noise was added to distance and velocity sensors, showing only minor performance degradation.
  • Efficiency: By enforcing permutation invariance, the researchers reduced the search space the model had to learn, leading to faster convergence and more stable behavior.

5. Conclusion and Future Work

DeepSet-Q effectively solves the fixed-input problem for autonomous driving agents, offering a scalable and robust alternative to traditional grid-based or neighbor-list methods. While the current model focuses on passenger cars, future work aims to include heterogeneous sets with pedestrians, cyclists, and traffic signals. Additionally, incorporating temporal context through recurrent units could further improve the agent's ability to predict the behavior of surrounding actors. Moving from simulation to real-world deployment with noisy sensor data remains the next major hurdle for this promising architecture.

Summary updated

Abstract

In many real-world decision making problems, reaching an optimal decision requires taking into account a variable number of objects around the agent. Autonomous driving is a domain in which this is especially relevant, since the number of cars surrounding the agent varies considerably over time and affects the optimal action to be taken. Classical methods that process object lists can deal with this requirement. However, to take advantage of recent high-performing methods based on deep reinforcement learning in modular pipelines, special architectures are necessary. For these, a number of options exist, but a thorough comparison of the different possibilities is missing. In this paper, we elaborate limitations of fully-connected neural networks and other established approaches like convolutional and recurrent neural networks in the context of reinforcement learning problems that have to deal with variable sized inputs. We employ the structure of Deep Sets in off-policy reinforcement learning for high-level decision making, highlight their capabilities to alleviate these limitations, and show that Deep Sets not only yield the best overall performance but also offer better generalization to unseen situations than the other approaches.
BibTeX
@inproceedings{hugle2019dynamic,
  author = {Maria Hügle and Gabriel Kalweit and Branka Mirchevska and Moritz Werling and Joschka Boedecker},
  title = {{Dynamic Input for Deep Reinforcement Learning in Autonomous Driving}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=3PZ9aX6plGw},
}
Figure