Delay-Aware Multi-Agent Reinforcement Learning for Cooperative Adaptive Cruise Control with Model-Based Stability Enhancement

Jiaqi Liu, Ziran Wang, Peng Hang, Jian Sun

Abstract

Cooperative Adaptive Cruise Control (CACC) is essential for the deployment of Connected and Automated Vehicles (CAVs) in platoons, offering benefits in road capacity and fuel efficiency. However, real-world communication and sensor delays often lead to string instability and safety hazards. This paper presents DAMARL (Delay-Aware Multi-Agent Reinforcement Learning), a framework designed to maintain platoon stability and safety under significant delays. By integrating a delay-aware Markov decision process with a physics-based action filter, DAMARL ensures robust performance where traditional RL methods fail.

Introduction

In the pursuit of intelligent transportation systems, Cooperative Adaptive Cruise Control (CACC) allows vehicles to travel in tight formations, or platoons. This coordination relies heavily on Vehicle-to-Vehicle (V2V) communication and real-time sensor data. While Reinforcement Learning (RL) has shown promise in managing the non-linear dynamics of these systems, most models assume ideal conditions. In practice, delays—stemming from communication latency, sensor processing, and mechanical lag—are inevitable. These delays can cause small speed fluctuations to amplify as they propagate through the platoon, a phenomenon known as string instability, which often results in rear-end collisions.

Methodology

The DAMARL framework addresses the challenge of delays through a three-pronged approach:

1. Restoring the Markov Property (MADA-MDP)

Standard RL relies on the Markov property, where the next state depends only on the current state and action. Delays break this assumption because the environment's current state is a result of actions initiated several time steps ago. DAMARL introduces the Multi-Agent Delay-Aware Markov Decision Process (MADA-MDP), which augments the state space with a buffer of "planned but not yet executed" actions. This memory allows the agent to understand its pending trajectory and make informed decisions despite the lag.

2. Attention-Integrated Policy

In a multi-agent platoon, the relevance of information varies. DAMARL utilizes an attention mechanism that allows each vehicle to dynamically weight the importance of data received from its neighbors. This ensures that critical information, such as the behavior of the immediate leader, is prioritized over less relevant data from distant vehicles, reducing the impact of noise in V2V communication.

3. Model-Based Action Filter

To bridge the gap between data-driven AI and control theory, DAMARL incorporates a physics-based safety net. The raw acceleration commands generated by the RL policy are passed through a velocity optimization model. If the RL suggests an action that would jeopardize safety or string stability, the filter adjusts the command to a safer, mathematically optimized acceleration value.

The system is trained using Centralized Training with Decentralized Execution (CTDE), allowing agents to learn from global information while operating independently during deployment.

Experimental Setup

The researchers evaluated DAMARL in a simulation environment configured with a 0.5-second total delay—a realistic but high-stress value for autonomous systems. Two primary scenarios were tested:

  • Catchup Scenario: Platoon members must accelerate from a standstill or lower speed to catch a distant lead vehicle.
  • Slowdown Scenario: The platoon must execute a coordinated high-speed deceleration.

Tests were conducted with platoons of 5, 8, and 10 vehicles to assess how the coordination difficulty scales with fleet size.

Results and Discussion

DAMARL was benchmarked against three prominent algorithms: MADDPG, ConseNet, and NeurComm. The results highlight the necessity of delay-aware architectures:

  • Safety and Reliability: In the challenging "Catchup" scenario, DAMARL and ConseNet achieved zero collisions. In contrast, MADDPG and NeurComm resulted in 27 and 50 collisions, respectively, demonstrating that standard RL cannot safely navigate significant communication lags.
  • String Stability: DAMARL reached target velocities and headways faster than all baselines, exhibiting minimal oscillation. The model-based action filter effectively suppressed the propagation of speed fluctuations.
  • Scalability: As the number of vehicles increased to 10, the performance of baseline methods degraded significantly. DAMARL maintained high rewards and stability, proving its robustness for larger-scale deployments.
  • Learning Efficiency: The inclusion of the MADA-MDP structure allowed the model to converge to a stable policy faster than the comparison algorithms.

Conclusion

DAMARL demonstrates that the integration of delay-aware mathematical modeling and physics-based constraints can overcome the inherent limitations of pure Reinforcement Learning in safety-critical tasks. While this study focused on longitudinal control, it provides a foundation for future research into lateral maneuvers (lane changes) and mixed-traffic scenarios where CAVs must interact with human-driven vehicles. By explicitly accounting for system delays, DAMARL moves CACC technology closer to safe, real-world implementation.

Summary updated

BibTeX
@inproceedings{liu2024delay,
  author = {Jiaqi Liu and Ziran Wang and Peng Hang and Jian Sun},
  title = {{Delay-Aware Multi-Agent Reinforcement Learning for Cooperative Adaptive Cruise Control with Model-Based Stability Enhancement}},
  booktitle = {ITSC},
  year = {2024},
  doi = {10.1109/itsc58415.2024.10919632},
}
Figure