Tactical Decision Making for Autonomous Trucks by Deep Reinforcement Learning with Total Cost of Operation Based Reward

Deepthi Pathare, Leo Laine, Morteza Haghir Chehreghani

Abstract

Autonomous trucking presents unique challenges due to vehicle size, limited maneuverability, and strict economic requirements. This paper introduces a hybrid Deep Reinforcement Learning (DRL) framework that separates high-level tactical decisions from low-level vehicle control. By utilizing a reward function based on the Total Cost of Operation (TCOP)—incorporating energy consumption, driver wages, and insurance risks—the system demonstrates improved safety and economic efficiency over traditional end-to-end reinforcement learning methods.

Introduction

While autonomous driving research has flourished for passenger vehicles, heavy-duty trucks require a specialized approach. Logistics companies operate on thin margins where every liter of fuel and every hour of driver time matters. Current RL research often suffers from two primary issues: inefficient low-level control leading to "jittery" behavior, and reward functions that fail to account for the complex economic trade-offs inherent in commercial transport.

Proposed Architecture

The researchers propose a hierarchical architecture that moves away from "end-to-end" control. This separation of concerns allows the RL agent to focus on strategy while a physics-based model handles vehicle dynamics.

  1. High-Level Agent (RL): Utilizing algorithms such as PPO, A2C, or DQN, this agent makes tactical decisions every second. These decisions include adjusting time gaps, modifying desired speeds, or initiating lane changes.
  2. Low-Level Controller: A physics-based Intelligent Driver Model (IDM) translates these tactical goals into specific longitudinal accelerations.
The interaction loop between the RL agent and the SUMO simulation environment.

Simulation Environment and State Space

The study utilizes the SUMO (Simulation of Urban Mobility) suite to model a dynamic highway environment. The ego-truck operates alongside 15 passenger cars, navigating based on a defined sensor range and relative distances to surrounding vehicles.

The highway simulation environment showing the ego-truck, passenger cars, and the sensor range used for state representation.

TCOP-Based Reward Function

To align the agent's behavior with business objectives, a Total Cost of Operation (TCOP) reward function was developed. This function calculates costs in Euros, factoring in:

  • Energy Consumption: Based on vehicle mass, aerodynamic drag, and rolling resistance.
  • Driver Wages: Calculated as a per-hour cost.
  • Insurance and Risk: Penalties for collisions or near-miss scenarios.
  • Revenue: A positive reward granted upon reaching the destination.

A critical finding was that normalizing these costs by distance traveled was essential for policy convergence, as it balanced the conflicting goals of speed (reducing wage costs) and efficiency (reducing energy costs).

Key Results

The hierarchical RL approach significantly outperformed pure RL baselines across several metrics:

  • Reliability: The success rate reached 97.8%, compared to 70.6% for the baseline.
  • Safety: Collision rates dropped from 29.4% to a mere 1.6%.
  • Economic Performance: The agent achieved a total operation cost of 3.74 Euros for the test route, outperforming SUMO’s default human-like driving models (3.85 Euros).
  • Algorithm Robustness: Proximal Policy Optimization (PPO) proved to be the most consistent algorithm for these tactical tasks.

Conclusion and Future Work

By decoupling tactical strategy from physical control and grounding rewards in operational costs, autonomous trucks can achieve a superior balance of safety and profitability. Future research will focus on more complex scenarios, such as highway on-ramps and steep grades, and exploring transfer learning to adapt these models to diverse road environments.

Summary updated

BibTeX
@article{pathare2026tactical,
  author = {Deepthi Pathare and Leo Laine and Morteza Haghir Chehreghani},
  title = {{Tactical Decision Making for Autonomous Trucks by Deep Reinforcement Learning with Total Cost of Operation Based Reward}},
  journal = {Artif. Intell. Rev.},
  year = {2026},
  doi = {10.1007/s10462-025-11448-8},
}
Figure