Communication-Efficient MARL for Platoon Stability and Energy-Efficiency Co-Optimization in Cooperative Adaptive Cruise Control of CAVs

Min Hua, Dong Chen, Kun Jiang, Fanggang Zhang, Jinhai Wang, Bo Wang, Quan Zhou, Hongming XuIEEE Trans. Vehicular Technology 2024 · arXiv

TL;DR

This paper introduces a communication-efficient Multi-Agent Reinforcement Learning (MARL) framework designed to help autonomous vehicle platoons balance two often-conflicting goals: string stability (safety/spacing) and energy efficiency. By combining quantized gradient descent with a novel "Binary Differential Consensus" method, the researchers achieved up to 5.8% energy savings while significantly reducing the data bandwidth required for vehicles to coordinate.

What Problem Does This Paper Address?

Cooperative Adaptive Cruise Control (CACC) allows Connected and Autonomous Vehicles (CAVs) to travel in tight "platoons," reducing air resistance and improving traffic flow. However, two major hurdles remain for real-world deployment:

  1. The Optimization Trade-off: It is computationally difficult to optimize for both platoon stability (keeping consistent gaps to avoid "slinky" effects) and energy efficiency (minimizing harsh acceleration/braking) simultaneously.
  2. Communication Constraints: Most MARL models assume perfect, high-bandwidth communication. In reality, vehicle-to-vehicle (V2V) networks have limited bandwidth and suffer from delays. Centralized controllers are also risky, as a single failure can compromise the whole platoon.

The authors tackle these by proposing a fully decentralized framework where vehicles learn to coordinate using minimal data exchange.

Methods

The authors developed a decentralized MARL architecture where each vehicle acts as an independent agent. Key technical components include:

  • Multi-Objective Reward Function: The agents are trained to minimize five factors: spacing error, velocity error, acceleration (for comfort), a safety penalty for near-collisions, and—crucially—electrical power consumption.
  • QSGD (Quantized Stochastic Gradient Descent): To save bandwidth, instead of sending high-precision floating-point numbers over the network, the system quantizes gradients into just three values (-1, 0, 1).
  • BDC (Binary Differential Consensus): This is the paper's core innovation. It’s a communication protocol where agents update their internal neural network weights based on the quantized difference between their weights and those of their neighbors. This allows the fleet to reach a "consensus" on driving strategy without a central server.
  • LSTM Integration: The actor and critic networks use Long Short-Term Memory (LSTM) layers to better process the temporal sequences of driving data.

Datasets

  • Simulated Training: The agents were trained in a custom environment modeled on plug-in hybrid vehicle dynamics.
  • OpenACC: For real-world validation, the authors used the Open Adaptive Cruise Control (OpenACC) database from the European Commission’s Joint Research Centre. This dataset contains sensor data from 10 commercially available vehicles with tunable ACC settings, providing a realistic benchmark for velocity profiles and perturbations.

Key Insights and Ideas

  • Less is More in Communication: The study proves that high-fidelity data exchange isn't strictly necessary for coordination. By using BDC-MARL, vehicles can synchronize their behavior effectively even when communication is "compressed" to binary-style signals.
  • Decentralization Equals Robustness: Because the model relies on local neighbor information rather than a global state, the failure of one vehicle's communication doesn't cause the entire system to collapse.
  • The "Sweet Spot" for Platoon Size: Through scalability tests, the researchers found that a 6-vehicle platoon is the most advantageous. While larger platoons benefit more from aerodynamic drafting, they become exponentially harder to stabilize. A size of 6 offered the best balance of fuel economy and operational stability.

Key Results

  • Energy Efficiency: The proposed BDC-MARL algorithm achieved the highest energy savings among all tested methods, improving efficiency by up to 5.8%.
  • Superior Stability: Compared to other popular MARL algorithms like IA2C and CommNet, BDC-MARL showed much smoother convergence. It maintained an average inter-vehicle spacing of 20.76 meters with significantly less "oscillation" in velocity and acceleration.
  • Bandwidth Efficacy: The system maintained high performance despite the quantized (low-bit) communication, effectively mitigating the delays that typically hinder real-time CACC tasks.

Limitations and Future Work

  • Short-Term Analysis: The energy consumption validation was conducted over relatively short driving cycles. The authors note that longer-term studies are needed to fully assess the cumulative energy-saving potential.
  • Homogeneity Assumption: The current model assumes vehicles have identical physical characteristics. Future work could explore "heterogeneous" platoons consisting of different vehicle types (e.g., a mix of trucks and passenger cars).
  • Complex Maneuvers: The study focuses on longitudinal control (speed and spacing). Integrating lateral control (lane changing) into this communication-efficient framework is a logical next step.

Summary updated

Abstract

Cooperative adaptive cruise control (CACC) is a critical function that faces significant challenges in maintaining platoon stability and achieving energy efficiency, especially in real-world operations. The CACC of connected and autonomous vehicles (CAVs) based on the multi-agent reinforcement learning (MARL) algorithm is studied to optimize platoon stability and energy efficiency simultaneously. Then the effectiveness of communication information is the key to guaranteeing learning performance in real-world driving, and thus this paper has proposed a communication-efficient MARL by incorporating the quantified stochastic gradient descent (QSGD) and a binary differential consensus (BDC) method into a fully-decentralized MARL framework. We evaluate this BDC-MARL algorithm against several typical non-communicative and communicative MARL algorithms, including IA2C, FPrint, and DIAL, focusing on metrics such as platoon stability, fuel economy, and driving comfort. Our results demonstrate that BDC-MARL achieves superior energy savings, with improvements of up to 5.8%, an average velocity of 15.26 m/s, and an inter-vehicle spacing of 20.76 m. Additionally, we perform comprehensive analyses of communicative information-sharing efficiency and scalability across varying platoon sizes, further validating the practical effectiveness through real-world scenarios using data from the open-source OpenACC.
BibTeX
@article{hua2024communication,
  author = {Min Hua and Dong Chen and Kun Jiang and Fanggang Zhang and Jinhai Wang and Bo Wang and Quan Zhou and Hongming Xu},
  title = {{Communication-Efficient MARL for Platoon Stability and Energy-Efficiency Co-Optimization in Cooperative Adaptive Cruise Control of CAVs}},
  journal = {IEEE Transactions on Vehicular Technology},
  year = {2024},
}
Figure