Abstract
The common pipeline in autonomous driving systems is highly modular and includes a perception component which extracts lists of surrounding objects and passes these lists to a high-level decision component. In this case, leveraging the benefits of deep reinforcement learning for high-level decision making requires special architectures to deal with multiple variable-length sequences of different object types, such as vehicles, lanes or traffic signs. At the same time, the architecture has to be able to cover interactions between traffic participants in order to find the optimal action to be taken. In this work, we propose the novel Deep Scenes architecture, that can learn complex interaction-aware scene representations based on extensions of either 1) Deep Sets or 2) Graph Convolutional Networks. We present the Graph-Q and DeepScene-Q off-policy reinforcement learning algorithms, both outperforming state-of-the-art methods in evaluations with the publicly available traffic simulator SUMO.
Introduction
Modern autonomous driving (AD) systems typically rely on a modular pipeline where a perception layer identifies environmental objects and passes them to a decision-making layer. However, applying Deep Reinforcement Learning (DRL) to this decision-making stage presents significant challenges. Traditional neural networks, such as Multi-Layer Perceptrons (MLPs), require fixed-size inputs, yet the number of surrounding vehicles, lanes, and traffic signs is inherently dynamic and variable.
While "Deep Sets" architectures can handle variable-length lists, they often treat objects as independent entities and struggle with heterogeneous data types. In realistic driving scenarios, safety and efficiency depend on understanding complex interactions—such as how a vehicle in an adjacent lane affects a merge maneuver. This paper introduces Deep Scenes, an architecture designed to bridge this gap by enabling agents to process diverse, variable-count objects while explicitly modeling their interactions.
Methodology: The Deep Scenes Architecture
The authors propose three primary architectural contributions to handle the complexity of traffic scenes:
- Deep Scene-Sets: This extends the Deep Sets architecture by using multiple specialized encoder modules ($\phi^k$) to process different object types (e.g., vehicles, lanes, signs) into a unified "encoded object space." These representations are pooled and processed by a secondary network ($\rho$) to generate a fixed-size "encoded scene" vector.
- Graph-Q: A DRL algorithm utilizing Graph Convolutional Networks (GCNs) to represent the state. By treating vehicles as nodes and their spatial relationships as edges, the network can reason about mutual influences through message passing.
- Deep Scene-Graphs: A hybrid approach that combines GCNs for modeling dynamic vehicle interactions with the multi-object logic of Deep Scene-Sets to incorporate static features like lane geometry.
Experimental Setup
The architectures were evaluated using the SUMO (Simulation of Urban MObility) simulator across two primary scenarios:
- Highway: A 1000m circular track with three lanes and varying vehicle densities.
- Fast Lanes: A complex environment featuring variable lane counts, diverse vehicle types (cars, trucks, motorcycles), and traffic signs indicating lane changes.
Training utilized datasets of 500,000 transitions collected via a rule-based "teacher" policy to facilitate off-policy learning.
Key Insights and Results
Interaction and Density
GCN-based models significantly outperformed set-based methods in medium-to-high traffic densities. While simpler architectures perform adequately in sparse traffic where interactions are minimal, the ability of GCNs to model "second-order" interactions (how a neighbor reacts to their own neighbor) becomes critical as congestion increases.
Architectural Robustness
A key finding was the benefit of parameter sharing within the object encoders ($\phi$). Sharing parameters across the final layers of these encoders improved both performance and robustness, suggesting that the model learns a more generalized representation of traffic participants.
Performance Benchmarks
- Superiority over Baselines: Graph-Q outperformed state-of-the-art methods like SocialCNN and VBIN (Vehicle Behavior Interaction Networks). Unlike VBIN, which is often restricted to a fixed number of neighbors, Deep Scenes maintains performance regardless of the number of surrounding objects.
- Statistical Significance: T-tests confirmed that the performance advantage of Graph-Q over DeepSet-Q is highly significant ($p < 0.001$).
- Behavioral Adaptation: In the "Fast Lanes" scenario, the DeepScene-Q agent learned to utilize fast lanes 39% of the time to maintain target speeds, a behavior rarely exhibited by standard rule-based controllers.
Conclusion and Future Work
The Deep Scenes architecture demonstrates that combining GCNs with flexible set-based encoding allows DRL agents to navigate complex, heterogeneous traffic environments more effectively than traditional modular approaches. While current models use fixed heuristics for graph construction, future research will explore dynamic graph discovery to let models learn which connections are most relevant for decision-making. Additionally, integrating temporal context via Transformers or Recurrent Neural Networks (RNNs) remains a promising avenue for improving long-term planning in dynamic scenes.
Summary updated