Abstract
Autonomous driving requires agents to navigate environments with a highly variable number of actors. Traditional Deep Reinforcement Learning (DRL) models often rely on fixed-size inputs, such as occupancy grids or $n$-nearest neighbor lists, which limit flexibility and generalization. This paper introduces DeepSet-Q, an architecture that utilizes "Deep Sets" to process a variable number of surrounding vehicles. The approach is permutation-invariant and demonstrates significantly better performance and robustness in dense, unseen traffic scenarios compared to standard fixed-input baselines.
1. Introduction: The Fixed-Input Bottleneck
In autonomous driving, the environment is inherently dynamic. An agent may be alone on a highway one moment and surrounded by dozens of vehicles the next. Most DRL agents are constrained by fixed-size input layers, forcing researchers to adopt suboptimal "hacks." One common approach is the $n$-nearest neighbor method, which only considers a fixed number of nearby cars. This can leave the agent "blind" to critical vehicles just outside the selection. Another approach, Occupancy Grids, represents the world as a top-down pixelated grid. However, these are computationally expensive and limited by predefined boundaries.
Furthermore, standard neural networks are sensitive to the order of inputs. Swapping the positions of two vehicles in an input list can result in different actions, even if the physical state remains identical. This paper addresses these challenges by proposing a method that is both flexible in input size and permutation-invariant.
2. The DeepSet-Q Architecture
The researchers from the University of Freiburg and the BMW Group propose DeepSet-Q, which integrates the Deep Sets architecture into a Deep Q-Network (DQN) framework. Unlike traditional models that use a single large vector, DeepSet-Q treats the input as a set of feature vectors, one for each detected vehicle.
As shown in the architecture above, each vehicle's data is processed by the same small network ($\phi$). These outputs are then combined using a symmetric pooling operation (such as a sum), ensuring that the order of vehicles in the list does not affect the outcome. Finally, a second network ($\rho$) processes this combined representation to determine the optimal driving action. This structure allows the model to learn universal features of "car-ness" that apply regardless of the total number of vehicles present.
3. Experimental Methodology
The study utilized the SUMO (Simulation of Urban MObility) traffic simulator to create a 1000m circular highway environment with 3 to 5 lanes. The agents were tested on various scenarios, including standard highway driving and complex merging tasks.
To test generalization, the researchers trained the models on a dataset containing at most six surrounding vehicles and then evaluated them in dense traffic scenarios with up to 90 vehicles. This "Generalization Test" aimed to determine if an agent trained on simple traffic could handle crowded highways without additional training.
4. Key Results and Insights
DeepSet-Q achieved the highest mean reward and lowest variance across all tested traffic densities. Its primary advantage lies in structural generalization: because the $\phi$ network is applied identically to every car, the model understands the concept of a vehicle as an object rather than a fixed coordinate in a grid.
- Superior Generalization: While CNN-based occupancy grids and fixed-input models saw performance plummet when faced with more vehicles than seen during training, DeepSet-Q maintained high performance.
- Robustness to Noise: The model remained robust even when Gaussian noise was added to distance and velocity sensors, showing only minor performance degradation.
- Efficiency: By enforcing permutation invariance, the researchers reduced the search space the model had to learn, leading to faster convergence and more stable behavior.
5. Conclusion and Future Work
DeepSet-Q effectively solves the fixed-input problem for autonomous driving agents, offering a scalable and robust alternative to traditional grid-based or neighbor-list methods. While the current model focuses on passenger cars, future work aims to include heterogeneous sets with pedestrians, cyclists, and traffic signals. Additionally, incorporating temporal context through recurrent units could further improve the agent's ability to predict the behavior of surrounding actors. Moving from simulation to real-world deployment with noisy sensor data remains the next major hurdle for this promising architecture.
Summary updated