Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning

Mohamed K. Abdel-Aziz, Cristina Perfecto, Sumudu Samarakoon, Mehdi Bennis, Walid Saad

Abstract

Cooperative perception plays a vital role in extending a vehicle’s sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared? at which resolution? and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperative perception messages (CPMs) by utilizing a quadtree-based point cloud compression mechanism. Furthermore, a federated RL approach is introduced in order to speed up the training process across vehicles. Simulation results show the ability of the RL agents to efficiently learn the vehicles’ association, RB allocation, and message content selection while maximizing vehicles’ satisfaction in terms of the received sensory information. The results also show that federated RL improves the training process, where better policies can be achieved within the same amount of time compared to the non-federated approach.

Introduction

Autonomous and connected vehicles rely on sensors like LiDAR and cameras, but these are limited by physical obstructions (buildings, other trucks) and weather. Cooperative perception—where vehicles swap sensor data—can solve this, but there is a major catch: raw sensor data (like LiDAR point clouds) is massive. Broadcasting everything to everyone would instantly crash the wireless network. Therefore, vehicles must decide who to pair with, which radio resources to use, and what specific parts of their sensor map to send at what resolution.

System Model and Problem Formulation

The proposed framework addresses these challenges by breaking the problem into two scales. At the RSU Level (Coarse Scale), a Road-Side Unit (RSU) acts as a high-level manager, using Reinforcement Learning (RL) to pair vehicles and allocate radio resources for a set time frame. At the Vehicle Level (Fine Scale), individual vehicles use RL to select which "blocks" of their sensor data to send during each time slot.

System model for vehicular cooperative perception at an intersection.

A key metric used is "vehicular satisfaction," which is based on the Age of Information (AoI)—how fresh the data is—and the Region of Interest (RoI)—how relevant the data is to the vehicle's current path and speed. As shown in the system model, vehicles must coordinate their transmissions within an RSU's coverage area to optimize these metrics under bandwidth constraints.

Proposed Methodology

To make this complex decision-making process feasible, the authors employ three key technical innovations:

  1. Quadtree Decomposition: Instead of raw points, they represent the world as a 2D grid that recursively divides into smaller blocks. This allows a vehicle to send a "low-res" block for an empty field but a "high-res" block for a crowded intersection.
  2. Branching Dueling Q-Network (BDQ): Traditional Deep RL fails when there are millions of possible actions. BDQ "branches" the decision-making process, allowing the agent to choose sub-actions independently but in a coordinated way. This turns an exponential combinatorial problem into a linear one.
  3. Federated Reinforcement Learning (FRL): Instead of every car learning in isolation, vehicles periodically share their neural network weights with the RSU. The RSU averages these weights and sends the "global" model back, allowing all vehicles to benefit from each other's experiences without sharing private raw data.

Evaluation and Results

The study uses SUMO (Simulation of Urban MObility) to simulate a traffic-light-regulated junction with 30 diverse vehicles. The RL agents were trained over 30,000 time slots using practical mobility traces.

  • Scalability: When the action space exploded to ~2 million actions at higher map resolutions, standard DQN failed to train, while the BDQ agent continued to improve, demonstrating its ability to handle high-dimensional action spaces.
  • Training Efficiency: The Federated RL approach achieved significantly higher rewards in the same amount of training time compared to non-federated approaches, proving that model averaging accelerates the convergence of data-sharing policies.
  • Performance vs. Oracle: The trained RL agents were compared against a theoretical "Oracle" solver. The gap between the RL agent and the Oracle was remarkably small, and the agents consistently outperformed random baselines across various vehicle densities.

Conclusion and Future Work

This research provides a scalable framework for vehicular cooperative perception by combining action branching with federated learning. While the current model uses a 2D Quadtree, future work could extend this to Octrees for 3D LiDAR data. Additionally, exploring multi-RSU scenarios and the impact of sensor noise would further enhance the real-world applicability of this cooperative sensing paradigm.

Summary updated

Abstract

Cooperative perception plays a vital role in extending a vehicle's sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared?, at which resolution?, and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperative perception messages (CPMs) by utilizing a quadtree-based point cloud compression mechanism. Furthermore, a federated RL approach is introduced in order to speed up the training process across vehicles. Simulation results show the ability of the RL agents to efficiently learn the vehicles' association, RB allocation, and message content selection while maximizing vehicles' satisfaction in terms of the received sensory information. The results also show that federated RL improves the training process, where better policies can be achieved within the same amount of time compared to the non-federated approach.
BibTeX
@article{abdelaziz2022vehicular,
  author = {Mohamed K. Abdel-Aziz and Cristina Perfecto and Sumudu Samarakoon and Mehdi Bennis and Walid Saad},
  title = {{Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning}},
  journal = {IEEE Trans. Communications},
  year = {2022},
  doi = {10.1109/tcomm.2021.3126650},
}
Figure