Introduction
Navigating unsignaled intersections remains one of the most complex challenges for autonomous vehicles. These scenarios require agents to predict driver intent, manage conflicting objectives like safety and traffic flow, and handle partial observability caused by occlusions. Traditional industry standards rely on rule-based heuristics, such as Time-to-Collision (TTC). However, TTC assumes other vehicles move at constant velocities and requires full environmental visibility. In dense or occluded traffic, TTC often becomes overly conservative, leading to "freezing" or failure. This research investigates whether Deep Reinforcement Learning (RL) can develop more adaptable policies, including active sensing behaviors like "creeping" forward to navigate these complex environments.
Methodology
The researchers framed intersection navigation as a Markov Decision Process (MDP) solved using Deep Q-Networks (DQNs). To understand how control granularity affects learning, they tested three action representations:
- Sequential Actions: Continuous control over acceleration and velocity at every timestep.
- Time-to-Go: A simplified binary choice between "waiting" or "going," where the agent commits to crossing once "go" is selected.
- Creep-and-Go: A hybrid approach allowing the agent to wait, move forward slowly (creep), or commit to crossing.
For occluded scenarios, the team utilized a dueling DQN architecture with prioritized experience replay. They integrated an occlusion indicator into the state grid—a discretized bird's-eye view of the environment—using a ray-tracing approach to represent blind spots.
Experimental Setup
Experiments were conducted using the SUMO (Simulation of Urban MObility) package. The authors designed five distinct intersection scenarios to test the agent's versatility: Right turn, Left turn, Left turn across two lanes (Left2), Forward crossing, and a high-density six-lane "Challenge" scenario. These were further modified with random visual occlusions to test the agent's ability to handle partial observability.
Key Insights and Results
The study revealed that action representation significantly impacts performance. Restricting the action space (as in Time-to-Go) actually facilitated the learning of more efficient departure times. Most notably, the RL agents naturally discovered "active sensing" behaviors. In occluded environments, the Creep-and-Go model learned to inch forward to improve its vantage point before committing to the intersection, a behavior vital for safety that traditional TTC lacks.
Performance Metrics
- Efficiency: The DQN Time-to-Go model was approximately 28% faster at reaching the goal than the TTC baseline.
- Dense Traffic: In the 6-lane Challenge scenario, TTC succeeded only 39.2% of the time due to excessive caution. The DQN model achieved a 98.46% success rate.
- Occlusions: In scenarios with 90-meter occlusions, the DQN-Creep model achieved a 96.3% success rate, significantly outperforming a creeping-augmented TTC (87.6%).
The Safety-Efficiency Trade-off
While the DQN models dominated the Pareto curve for speed and success rates, they introduced a critical trade-off: safety. While TTC maintained a 0.0% collision rate in standard scenarios, DQN models exhibited non-zero collision rates ranging from 0.01% to 1.37%. This highlights the primary hurdle for real-world deployment: the difficulty of guaranteeing absolute safety through standard reward shaping.
Conclusion and Future Work
Deep Reinforcement Learning demonstrates superior efficiency and problem-solving capabilities in complex, occluded intersections compared to rule-based heuristics. However, the occasional collisions remain unacceptable for production environments. Future research will focus on integrating hard safety constraints, such as shielding or formal "Safe RL" methods, to ensure zero-collision performance without sacrificing the efficiency gains provided by deep learning.
Summary updated