Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

Hooman Hamze, Benjamin Houshmand, Deepthi Pathare, Erik Börve, Morteza Haghir Chehreghani, Nikolce MurgovskiarXiv 2026 · arXiv

TL;DR

This paper introduces a hybrid control framework for autonomous trucks that combines the safety guarantees of Model Predictive Control (MPC) with the strategic foresight of Reinforcement Learning (RL). By using a pre-trained Deep Q-Network (DQN) to estimate the "terminal cost" (long-term value) of a state, the authors enable short-horizon MPC to make safer, more efficient decisions in interactive highway traffic without the heavy computational burden of long-horizon planning.

What Problem Does This Paper Address?

Controlling heavy-duty autonomous trucks on a highway is a balancing act. Because of their massive weight and limited maneuverability, trucks need to plan far ahead and cooperate with surrounding traffic to change lanes or merge safely.

Engineers typically use two tools, both with flaws:

  1. Model Predictive Control (MPC): Excellent at respecting physical constraints (like "don't hit that car"), but it is computationally expensive. To keep it fast, engineers use a "short horizon" (looking only a few seconds ahead). However, a short-sighted controller might drive into a trap because it doesn't know the long-term value of its final position.
  2. Reinforcement Learning (RL): Great at learning long-term strategies through trial and error, but it offers no safety guarantees. RL policies can be "jerky" or overly conservative, often failing to maintain smooth traffic flow.

The authors address this by using RL to "teach" the MPC what happens beyond its visible horizon.

Methods

The researchers developed a hybrid architecture where a finite-horizon MPC serves as the primary controller, but its "terminal cost"—the penalty or reward for the state it reaches at the end of its look-ahead—is provided by a pre-trained RL model.

Figure 1: Architecture of controller

Key components include:

  • MPC via Dynamic Programming (DP): The controller solves the driving problem over a discretized grid of speeds and lane-change actions.
  • Learned Terminal Cost: They trained a Deep Q-Network (DQN) offline. The Q-values represent the expected long-term cost from a given state. This value is plugged directly into the MPC's optimization.
  • Chance Constraints: To handle the real-world messiness of sensor noise and unpredictable neighbors, they used a stochastic formulation. This "inflates" safety margins based on how uncertain the truck is about a nearby vehicle's future position.
  • Trajectory Prediction: They found that a simple constant-acceleration model outperformed more complex reactive models for the short timescales (0.4s to 2.0s) used by the MPC.

Datasets

The framework was evaluated using SUMO (Simulation of Urban MObility), a popular open-source traffic simulator.

  • Scenario: A three-lane highway with high-density, reactive traffic. Surrounding vehicles were programmed to respond to the ego-truck's movements.
  • Training: The DQN was trained over millions of simulation steps to learn the cost-to-go for various traffic configurations.
  • Benchmarking: The hybrid model was compared against pure MPC (with no terminal cost), a "rollout" baseline (which simulates a simple heuristic policy forward), and a pure RL agent.

Key Insights and Ideas

  • The "Newton Step" Synergy: Theoretically, one step of MPC look-ahead on top of an approximate terminal cost acts like a Newton step for the Bellman equation. This means the MPC can "fix" the imperfections of a mediocre RL policy, leading to a much better result than either could achieve alone.
  • Short Horizons are Sufficient: With a learned terminal cost, a horizon of just 3 steps ($N=3$, or 0.6 seconds) outperformed a pure MPC looking 5 steps ahead. This significantly reduces the math the truck has to do every millisecond.
  • Rollouts are Overoptimistic: A common alternative to RL is "rollout"—simulating a simple base policy to estimate future costs. The authors found this performed poorly because simple policies can't capture the complex, interactive "dance" of highway traffic, leading the truck to take risks it shouldn't.

Key Results

The hybrid approach proved to be the "Goldilocks" solution:

  • Safety: The hybrid controller achieved zero collisions in stochastic testing. In contrast, pure MPC with a 1-step horizon crashed in 86% of episodes.
  • Efficiency: The $N=3$ hybrid configuration achieved the lowest total cost (329) of any safe configuration, beating pure MPC at $N=5$ (cost of 754).
  • Real-time Performance: The best-performing hybrid model ($N=3$) took roughly 91.6 ms per step, well within the 200 ms budget required for real-time implementation.

Detailed performance metrics can be seen in the cost decomposition table: Performance evaluation and cost decomposition

Table 1: Stochastic controller performance evaluation and detailed cost decomposition.

Limitations and Future Work

While promising, the study has a few constraints:

  • Simplified Geometry: The tests were conducted on straight, homogeneous highways. Real-world ramps, curves, and merges add significant complexity.
  • Kinematic Model: The truck model was kinematic, meaning it didn't account for complex dynamics like tire slip or suspension, which are critical for heavy vehicles at high speeds.
  • Lane-Change Prediction: The current model assumes surrounding vehicles stay in their lanes during the truck's look-ahead horizon. Future work could integrate more advanced "intent prediction" for neighboring drivers.

Summary updated

Abstract

Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.
BibTeX
@misc{hamze2026learned,
  author = {Hooman Hamze and Benjamin Houshmand and Deepthi Pathare and Erik Börve and Morteza Haghir Chehreghani and Nikolce Murgovski},
  title = {{Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic}},
  year = {2026},
  eprint = {2610.40257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40257},
}
Figure