Tactical decision making for autonomous trucks by deep reinforcement learning with total cost of operation based reward

Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

Introduction

Autonomous driving research has traditionally prioritized passenger vehicles, focusing primarily on safety and collision avoidance. However, heavy-duty trucks present a distinct set of challenges due to their significant mass, length, and the stringent economic constraints of the freight industry. In commercial trucking, a "safety-first" or "speed-first" approach is insufficient; the ultimate metric of success is the Total Cost of Operation (TCOP). A truck that navigates safely but inefficiently—consuming excessive fuel or incurring high labor costs—is not commercially viable.

This paper introduces a hybrid Deep Reinforcement Learning (DRL) framework designed to bridge the gap between high-level tactical maneuvering and the economic realities of heavy vehicle operation. By separating strategic decision-making from low-level physical control and utilizing a TCOP-based reward signal, the researchers achieved a 97.8% success rate in highway navigation while outperforming standard driving models in economic efficiency.

Methodology

The authors propose a hierarchical architecture that decouples the driving task into two distinct layers:

  1. Tactical Layer (DRL Agent): This layer employs algorithms such as Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), or Deep Q-Network (DQN) to make high-level decisions at 1-second intervals. These actions include adjusting the desired time gap with leading vehicles, modifying target speeds, or initiating lane changes.
  2. Operational Layer (Low-Level Controllers): Rather than the RL agent directly controlling actuators (steering and pedals), it communicates its "intent" to physics-based models. The Intelligent Driver Model (IDM) handles longitudinal control (Adaptive Cruise Control), while the LC2013 model manages lateral control (lane changes). This separation ensures smoother, more predictable behavior suitable for heavy vehicles.

TCOP Reward Function

To align the agent's behavior with commercial goals, a multi-objective TCOP reward function was developed. It incorporates energy costs (calculated via a force-based model), labor costs (driver wages per hour), and revenue earned upon delivery, while penalizing collisions and near-misses.

Experimental Setup

The study utilized a dynamic simulation environment built on SUMO (Simulation of Urban Mobility). The scenario involved a 3000m highway stretch featuring three lanes and 15 passenger cars exhibiting non-deterministic behaviors to challenge the ego-vehicle—a heavy truck combination.

Simulation environment showing the ego-truck, passenger cars, and sensor range across three lanes.

The authors have open-sourced their tailored RL environment on GitHub to facilitate further research into heavy vehicle reinforcement learning.

Key Results and Insights

The hybrid architecture demonstrated significant improvements over end-to-end control baselines:

  • Success and Safety: The PPO + IDM architecture reached the target successfully 97.8% of the time, with collision rates dropping to just 1.6% (compared to 70.6% success and 29.4% collisions for the baseline).
  • Economic Efficiency: The TCOP-optimized agent achieved a total operation cost of 3.74 euros for the test run, outperforming the default SUMO "Krauss" car-following model (3.85 euros).
  • Algorithm Performance: PPO consistently outperformed DQN and A2C in terms of reward stability and final success rates.

A key insight from the study is the "Conflict of Costs": driver costs incentivize speed, while energy costs incentivize slower, more efficient driving. The DRL agent successfully learned to balance these tensions, adopting sophisticated behaviors like drafting and strategic speed maintenance.

Conclusion and Future Work

By letting physics-based models handle the "smoothness" of control while the RL agent focuses on "strategy," the researchers achieved high stability and economic performance. While currently limited to flat highway scenarios, future work will incorporate diverse terrains (slopes), complex urban maneuvers (merging, roundabouts), and transfer learning to adapt agents to more challenging environments without retraining from scratch.

Summary updated

BibTeX
@article{pathare2026tactical,
  author = {Deepthi Pathare and Leo Laine and Morteza Haghir Chehreghani},
  title = {{Tactical decision making for autonomous trucks by deep reinforcement learning with total cost of operation based reward}},
  journal = {Artif. Intell. Rev.},
  year = {2026},
  doi = {10.1007/s10462-025-11448-8},
}
Figure