This paper, presented by researchers from the University of Freiburg and the BMW Group, tackles a fundamental bottleneck in applying Deep Reinforcement Learning (DRL) to autonomous driving: the "fixed-input" problem.
TL;DR
The researchers introduce DeepSet-Q, a reinforcement learning architecture that uses "Deep Sets" to process a variable number of surrounding vehicles. Unlike traditional models that rely on fixed-size grids or limited neighbor lists, DeepSet-Q is permutation-invariant and generalizes significantly better to complex, unseen traffic scenarios.
What Problem Does This Paper Address?
In autonomous driving, the environment is highly dynamic. At one moment, you might be alone on a highway; the next, you are surrounded by a dozen cars.
Most DRL agents are built with fixed-size input layers. To make this work, engineers usually employ one of two "hacks":
- The $n$-nearest approach: Only consider the $n$ closest cars. If the $(n+1)$-th car is the one about to cause a collision, the agent is effectively blind to it.
- Occupancy Grids: Represent the world as a top-down pixelated grid. This is computationally expensive, and if a car is outside the predefined grid boundaries, it doesn't exist to the agent.
Furthermore, standard neural networks are sensitive to the order of inputs. If you swap the positions of "Car A" and "Car B" in a list, a standard network might output a different action, even though the physical reality hasn't changed. This paper seeks a method that is both flexible in size and "permutation invariant."
Methods
The authors propose DeepSet-Q, integrating the "Deep Sets" architecture into a Deep Q-Network (DQN) framework.
- Deep Sets Architecture: Instead of a single large vector, the input is a set of feature vectors (one for each detected car).
- Each car's data is processed by the same small network ($\phi$).
- The outputs are then combined using a symmetric pooling operation (like a sum or max). This ensures that the order of cars in the list doesn't matter.
- A second network ($\rho$) processes this combined "set representation" to help decide the best driving action.
- Set2Set-Q: As a more complex baseline, they also implemented an architecture using an LSTM with an attention mechanism to process the set.
- Baselines: They compared these against a Relational Grid (fixed list of neighbors), a CNN (occupancy grid), and a standard Rule-based lane-change controller.
Datasets
The study used the SUMO (Simulation of Urban MObility) traffic simulator.
- Environment: A 1000m circular highway with 3 to 5 lanes.
- Dataset 1: 500,000 samples featuring an arbitrary number of surrounding vehicles.
- Dataset 2 (The Generalization Test): 500,000 samples containing at most six surrounding vehicles. This was used to see if an agent trained on simple traffic could handle dense, crowded highways.
Key Insights and Ideas
- Structural Generalization: The "aha moment" is that Deep Sets don't just handle variable inputs; they understand the concept of a vehicle. Because the $\phi$ network is applied to every car identically, the model learns universal features of "car-ness" that apply whether there are 2 cars or 20.
- The Grid Bottleneck: Occupancy grids (CNNs) struggle when the agent moves to a 5-lane highway if they were only trained on 3 lanes. Deep Sets don't care about the number of lanes; they only care about the objects within them.
- Permutation Invariance is Key: By forcing the model to be invariant to the order of vehicles, the researchers reduced the "search space" the model had to learn, leading to faster convergence and more stable behavior.
Key Results
- Top Performance: DeepSet-Q achieved the highest mean reward and the lowest variance across all tested traffic densities (30 to 90 vehicles in the scenario).
- Superior Generalization: When trained on Dataset 2 (max 6 cars) and tested on dense traffic, DeepSet-Q maintained high performance. In contrast, the CNN and Fixed-Input models saw their performance plummet because they couldn't process the "extra" vehicles they hadn't seen during training.
- Robustness: The authors added Gaussian noise to the distance and velocity sensors. DeepSet-Q remained robust, showing only a minor decrease in performance even with significant sensor noise.
Limitations and Future Work
- Object Diversity: The current model focuses primarily on other passenger cars. The authors note that future iterations should include heterogeneous sets containing pedestrians, cyclists, and traffic lights.
- Temporal Context: While the model handles the spatial set well, it doesn't explicitly model the history of those objects (e.g., is that car accelerating or braking?). Incorporating recurrent units (RNNs) into the $\phi$ network could address this.
- Sim-to-Real: The work is entirely simulation-based. Moving from the "perfect" object lists of SUMO to the noisy, occluded detections of real-world LiDAR and cameras remains a significant hurdle.
Summary updated