Introduction
In the development of Connected Autonomous Vehicles (CAVs), a common assumption is that connectivity naturally leads to cooperation. However, in decentralized systems where individual agents are programmed to maximize personal utility—such as speed or travel time—cooperation is not guaranteed. Standard Multi-Agent Reinforcement Learning (MARL) algorithms often struggle to incentivize agents to perform maneuvers that benefit the collective at a personal cost. This paper addresses the fundamental challenge of reallocating system rewards to ensure that self-interested agents are motivated to cooperate within a stable coalition.
Methodology
The proposed framework bridges MARL with Cooperative Game Theory by introducing a reward reallocation mechanism based on the Shapley value. The approach follows a structured process:
- Transferable Utility (TU) Game Formulation: The CAV environment is modeled as a game where the value of any coalition of vehicles can be quantified.
- Characteristic Function Definition: A function $v(s, a, C)$ represents the expected discounted total reward a coalition $C$ can achieve given the current state and actions.
- Shapley Value Reallocation: Instead of raw local rewards, the system calculates the Shapley value for each agent, assigning a "fair" payout based on their marginal contribution to all possible coalitions.
- Stability and the Core: The authors mathematically prove that if the game is "convex" (meaning the incentive to join a group increases with group size), the Shapley value allocation lies in the Core. This ensures no sub-group has an incentive to defect.
- Algorithmic Implementation: A modified Actor-Critic algorithm is employed, where a neural network approximates the characteristic function to compute the Shapley values that guide policy updates.
Experimental Setup
The researchers evaluated the framework using the CARLA simulator, focusing on a high-fidelity three-lane freeway scenario.
The setup involved a mix of 30 vehicles, including CAVs controlled by the proposed algorithm and human-driven vehicles managed by CARLA's autopilot. Input data consisted of simulated camera images and LIDAR point clouds, processed via ResNet-18 and PointPillars for robust object detection.
Results and Discussion
The Shapley-based method was benchmarked against state-of-the-art algorithms, including MADDPG, M3DDPG, and COMA.
Key findings include:
- Superior System Reward: The proposed method achieved a significantly higher mean episode system reward than all baselines, indicating more efficient overall traffic flow.
- Scalability in Mixed Traffic: The algorithm demonstrated strong performance across varying CAV penetration rates. Transitioning from 0% to 100% CAVs resulted in a 10% increase in average velocity and an 8% improvement in passenger comfort.
- Stability over COMA: Even at low penetration rates, the Shapley-based reallocation outperformed the Counterfactual Multi-Agent (COMA) algorithm in both speed and comfort metrics.
The success of this approach stems from the fact that fairness, as defined by the Shapley value (symmetry and additivity), stabilizes the learning process. By identifying the convex nature of CAV rewards, the authors provide a mathematical guarantee for stable cooperation.
Limitations and Future Work
While effective, the method exhibits slower initial convergence compared to some baselines, as agents must first learn an accurate estimation of the characteristic function. Additionally, while tested with 30 vehicles, the combinatorial complexity of calculating coalitions may pose challenges in extremely dense urban environments. Future research will explore scaling to massive swarms and applying this framework to other networked Cyber-Physical Systems, such as smart grids or industrial robotics.
Summary updated