Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways

Branka Mirchevska, Maria Hügle, Gabriel Kalweit, Moritz Werling, Joschka Boedecker

Abstract

Well-established optimization-based methods can guarantee an optimal trajectory for a short optimization horizon, typically no longer than a few seconds. As a result, choosing the optimal trajectory for this short horizon may still result in a sub-optimal long-term solution. At the same time, the resulting short-term trajectories allow for effective, comfortable and provable safe maneuvers in a dynamic traffic environment. In this work, we address the question of how to ensure an optimal long-term driving strategy, while keeping the benefits of classical trajectory planning. We introduce a Reinforcement Learning based approach that coupled with a trajectory planner, learns an optimal long-term decision-making strategy for driving on highways. By online generating locally optimal maneuvers as actions, we balance between the infinite low-level continuous action space, and the limited flexibility of a fixed number of predefined standard lane-change actions. We evaluated our method on realistic scenarios in the open-source traffic simulator SUMO and were able to achieve better performance than the 4 benchmark approaches we compared against, including a random action selecting agent, greedy agent, high-level, discrete actions agent and an IDM-based SUMO-controlled agent.

Introduction

Autonomous driving (AD) systems face a fundamental tension between high-level strategic foresight and low-level tactical execution. Classical optimization methods, such as Model Predictive Control (MPC), are excellent at generating smooth, safe trajectories for the immediate future. However, these methods are often "short-sighted," failing to account for long-term global rewards, such as slowing down momentarily to merge into a faster-moving lane later.

Conversely, Reinforcement Learning (RL) excels at long-term strategy but struggles with action space definition. Discrete action sets (e.g., "move left") are often too rigid for fluid traffic, while continuous action spaces (e.g., steering angles) are computationally expensive to optimize and slow to train. This paper introduces Amortized Q-learning with Model-based Action Proposals (AQL-MAP), a framework that bridges this gap by using a model-based planner to propose a finite set of safe, reachable "gaps" in traffic as the RL agent's action space.

The AQL-MAP Framework

The core of AQL-MAP is the synergy between a Trajectory Planner (TP) and a Deep Q-Learning agent. Instead of the agent deciding on raw control inputs, it selects from a set of high-level "options" provided by the TP.

System architecture showing the interaction between the Trajectory Planner (TP), the SUMO simulator, and the RL Agent.

As shown in the architecture above, the system operates as a Semi-Markov Decision Process (Semi-MDP). The TP analyzes the current state from the SUMO simulator to identify which gaps in surrounding traffic are physically reachable and safe within a 6-second horizon. These gaps are passed to the RL agent as action proposals. The agent then selects the optimal gap based on its learned Q-values. The TP executes the low-level controls to reach that gap, ensuring the maneuver remains comfortable and safe.

Action Abstraction: Driving via Gaps

By abstracting the action space into "gaps," the agent avoids the complexity of continuous control while maintaining more flexibility than standard discrete lane-change commands. A gap is defined as the space between two vehicles (or a vehicle and the horizon) in any given lane.

Representation of traffic gaps (dashed boxes) as potential actions for the autonomous agent.

To handle the varying number of vehicles and available gaps, the authors utilize a DeepSet-Q architecture. This neural network design ensures permutation invariance, meaning the agent's decision remains consistent regardless of the order in which surrounding vehicles are processed. This allows the model to generalize across different traffic densities and configurations.

Key Insights and Results

The researchers trained the agent using approximately 150 hours of driving data (500,000 transitions) on a simulated 3-lane highway. Evaluation against four benchmarks—including a greedy agent and a standard SUMO-IDM controller—revealed several key advantages:

  • Strategic Sacrifice: In dense traffic, AQL-MAP demonstrated the ability to perform prolonged braking maneuvers to fall back and merge into a gap that eventually led to a faster lane. Simple greedy agents failed these scenarios by only prioritizing immediate speed.
  • Safety by Construction: Because the RL agent can only choose from gaps pre-vetted by the trajectory planner, the system inherits the formal safety guarantees of classical optimization methods.
  • Efficiency: The "amortized" approach—evaluating only a finite set of proposed gaps rather than searching a continuous space—allowed the agent to learn effective strategies from offline data with significantly lower sample requirements than traditional continuous RL.

Conclusion

AQL-MAP successfully combines the reliability of classical trajectory planning with the strategic depth of Reinforcement Learning. While current limitations include a restriction on double-lane changes and a lack of social interaction terms (like courtesy) in the reward function, the framework provides a robust foundation for highway navigation. Future work will focus on incorporating multi-agent interactions and addressing the transition from simulation to real-world sensor noise.

Summary updated

Abstract

Well-established optimization-based methods can guarantee an optimal trajectory for a short optimization horizon, typically no longer than a few seconds. As a result, choosing the optimal trajectory for this short horizon may still result in a sub-optimal long-term solution. At the same time, the resulting short-term trajectories allow for effective, comfortable and provable safe maneuvers in a dynamic traffic environment. In this work, we address the question of how to ensure an optimal long-term driving strategy, while keeping the benefits of classical trajectory planning. We introduce a Reinforcement Learning based approach that coupled with a trajectory planner, learns an optimal long-term decision-making strategy for driving on highways. By online generating locally optimal maneuvers as actions, we balance between the infinite low-level continuous action space, and the limited flexibility of a fixed number of predefined standard lane-change actions. We evaluated our method on realistic scenarios in the open-source traffic simulator SUMO and were able to achieve better performance than the 4 benchmark approaches we compared against, including a random action selecting agent, greedy agent, high-level, discrete actions agent and an IDM-based SUMO-controlled agent.
BibTeX
@inproceedings{mirchevska2021amortized,
  author = {Branka Mirchevska and Maria Hügle and Gabriel Kalweit and Moritz Werling and Joschka Boedecker},
  title = {{Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=UTdHXmiVoly},
}
Figure