Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways

Branka Mirchevska, Maria Hügle, Gabriel Kalweit, Moritz Werling, Joschka Boedecker

TL;DR

This paper introduces Amortized Q-learning with Model-based Action Proposals (AQL-MAP), a framework that combines the long-term strategic foresight of Reinforcement Learning (RL) with the safety and comfort of classical trajectory planning. By using a model-based planner to propose a finite set of "reachable gaps" in traffic as actions, the agent avoids the pitfalls of both overly simplistic discrete action sets and overly complex continuous control.

What Problem Does This Paper Address?

Autonomous driving (AD) systems often struggle to balance high-level strategy with low-level execution.

  1. Classical Optimization (MPC): These methods are great at generating safe, smooth trajectories for the next few seconds, but they are "short-sighted" and often fail to make globally optimal long-term decisions (like slowing down now to catch a gap in a faster lane later).
  2. Reinforcement Learning (RL): RL can learn long-term strategies, but choosing the right "action space" is hard. Discrete actions (e.g., "move left") are often too rigid and unnatural. Continuous actions (e.g., steering angle) are too granular, making the learning process slow and the optimization step (finding the best action) computationally expensive.

The authors aim to find a "middle ground" that provides the flexibility of continuous values with the efficiency of a finite action set.

Methods

The core innovation is AQL-MAP, which uses a trajectory planner to "propose" actions to an RL agent.

  • Action Abstraction (Gaps): Instead of steering angles, the agent's actions are "gaps"—the spaces between vehicles in surrounding lanes.
  • Model-based Proposals: A trajectory planner (TP) checks the environment and identifies which gaps are physically reachable and safe within a 6-second horizon. These become the "action proposals."
  • Amortized Q-learning: In standard continuous RL, finding the action that maximizes the Q-value is hard. Here, the "maximization" is simplified (amortized) by only evaluating the finite set of safe gaps proposed by the TP.
  • DeepSet-Q Architecture: To handle a varying number of surrounding cars and available gaps, the authors use a "DeepSet" architecture. This ensures the neural network is permutation-invariant (it doesn't matter what order the cars are listed in).
  • Options Framework: The system operates as a Semi-MDP. The agent picks a gap (an "option"), and the trajectory planner executes the low-level controls until the gap is reached or becomes unavailable.

Datasets

The researchers used the SUMO (Simulation of Urban MObility) traffic simulator.

  • Training Data: They collected ~150 hours of driving data (500,000 transitions) on a 3-lane highway. The environment featured 0 to 70 surrounding vehicles with randomized behaviors.
  • Evaluation: The agent was tested on 80 randomly generated scenarios with varying traffic densities (from light to heavy) and specific "critical" scenarios designed to test long-term planning.

Key Insights and Ideas

  • Safety by Construction: Because the RL agent can only choose from gaps that the trajectory planner has already deemed safe and reachable, the system inherits the safety guarantees of classical methods.
  • Strategic Sacrifice: One of the most interesting "aha" moments occurs in dense traffic. The AQL-MAP agent learned to perform a prolonged braking maneuver to fall back and merge into a gap that eventually leads to a faster lane. Simple greedy agents or discrete RL agents failed this test because they only looked at immediate speed gains.
  • Bridging the Representation Gap: By describing gaps with continuous features (distance, relative velocity) but treating the choice between them as a discrete selection, the authors effectively combine the best parts of discrete and continuous RL.

Key Results

  • Superior Speed: AQL-MAP achieved higher average speeds across all traffic densities compared to four benchmarks: a random agent, a greedy agent, a standard SUMO-IDM controller, and a high-level discrete RL agent.
  • Success in Critical Scenarios: In complex "bottleneck" situations where planning ahead is mandatory, AQL-MAP was often the only agent capable of successfully navigating to the optimal lane.
  • Efficiency: The "amortized" approach allowed the agent to learn effective strategies from offline data without the massive sample requirements usually seen in continuous-action RL.

Limitations and Future Work

  • Double Lane Changes: Currently, the trajectory planner does not consider moving across two lanes in one go for safety reasons. The authors note that this limits some maneuvers.
  • Social Interaction: The current reward function focuses on the agent's own speed and comfort. The authors suggest that future work should include "interaction terms" (like courtesy) to ensure the agent doesn't cut off other drivers or cause traffic waves.
  • Real-world Transfer: While SUMO is a standard tool, the transition from a 2D simulator to real-world sensor noise and vehicle dynamics remains a significant hurdle for any RL-based approach.

Summary updated

Abstract

Well-established optimization-based methods can guarantee an optimal trajectory for a short optimization horizon, typically no longer than a few seconds. As a result, choosing the optimal trajectory for this short horizon may still result in a sub-optimal long-term solution. At the same time, the resulting short-term trajectories allow for effective, comfortable and provable safe maneuvers in a dynamic traffic environment. In this work, we address the question of how to ensure an optimal long-term driving strategy, while keeping the benefits of classical trajectory planning. We introduce a Reinforcement Learning based approach that coupled with a trajectory planner, learns an optimal long-term decision-making strategy for driving on highways. By online generating locally optimal maneuvers as actions, we balance between the infinite low-level continuous action space, and the limited flexibility of a fixed number of predefined standard lane-change actions. We evaluated our method on realistic scenarios in the open-source traffic simulator SUMO and were able to achieve better performance than the 4 benchmark approaches we compared against, including a random action selecting agent, greedy agent, high-level, discrete actions agent and an IDM-based SUMO-controlled agent.
BibTeX
@inproceedings{mirchevska2021amortized,
  author = {Branka Mirchevska and Maria Hügle and Gabriel Kalweit and Moritz Werling and Joschka Boedecker},
  title = {{Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=UTdHXmiVoly},
}

Loading related content...

Figure