Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning

Mohamed K. Abdel-Aziz, Cristina Perfecto, Sumudu Samarakoon, Mehdi Bennis, Walid Saad

TL;DR

This paper proposes a novel framework for vehicular cooperative perception that allows cars to "see" beyond their line-of-sight by intelligently sharing sensor data. By combining Action Branching to handle massive decision spaces and Federated Reinforcement Learning (FRL) to speed up training, the system optimizes which data to share, at what resolution, and with which vehicles, all while staying within the limits of wireless bandwidth.

What Problem Does This Paper Address?

Autonomous and connected vehicles rely on sensors like LiDAR and cameras, but these are limited by physical obstructions (buildings, other trucks) and weather. Cooperative perception—where vehicles swap sensor data—can solve this, but there is a major catch: raw sensor data (like LiDAR point clouds) is massive.

Broadcasting everything to everyone would instantly crash the wireless network. Therefore, vehicles must decide:

  1. Who should I pair with to exchange data?
  2. Which radio resources (Resource Blocks) should be used?
  3. What specific parts of my sensor map should I send, and at what resolution?

Solving these three questions simultaneously is a "combinatorial nightmare" because the number of possible combinations of map pieces and vehicle pairs grows exponentially.

Methods

The authors break the problem into two scales:

  • RSU Level (Coarse Scale): A Road-Side Unit (RSU) acts as a high-level manager, using Reinforcement Learning (RL) to pair vehicles and allocate radio resources for a set "time frame."
  • Vehicle Level (Fine Scale): Individual vehicles use RL to select which "blocks" of their sensor data to send during each "time slot" within that frame.

To make this work, they employ three key technical innovations:

  1. Quadtree Decomposition: Instead of raw points, they represent the world as a 2D grid that recursively divides into smaller blocks. This allows a vehicle to send a "low-res" block for an empty field but a "high-res" block for a crowded intersection.
  2. Branching Dueling Q-Network (BDQ): Traditional Deep RL (like DQN) fails when there are millions of possible actions. BDQ "branches" the decision-making process, allowing the agent to choose sub-actions (like pairing for Vehicle A and pairing for Vehicle B) independently but in a coordinated way. This turns an exponential problem into a linear one.
  3. Federated Reinforcement Learning (FRL): Instead of every car learning in isolation, vehicles periodically share their neural network weights with the RSU. The RSU averages these weights and sends the "global" model back, allowing all vehicles to benefit from each other's experiences without sharing private raw data.

Datasets

The study uses SUMO (Simulation of Urban MObility), a standard open-source traffic simulation package.

  • Scenario: A traffic-light-regulated junction.
  • Scale: 30,000 time slots featuring 30 diverse vehicles (cars, buses, and trucks) entering and exiting the RSU's coverage area.
  • Data: Practical mobility traces were used to ensure the RL agents faced realistic vehicle densities and movement patterns.

Key Insights and Ideas

  • The "Satisfaction" Metric: The authors don't just measure data throughput. They define "vehicular satisfaction" based on the Age of Information (AoI)—how fresh the data is—and the Region of Interest (RoI)—how relevant the data is to the vehicle's current path and speed.
  • Scalability via Branching: The most significant "aha moment" is the use of BDQ. The paper shows that while a standard DQN agent is crushed by the computational weight of large map resolutions ($L=3$), the BDQ agent handles it with ease because its output layer grows linearly rather than combinatorially.
  • FRL as a Training Booster: Federated learning is often discussed for privacy, but here it is used as a performance booster. By averaging models, the agents converge on better data-sharing policies much faster than they would individually.

Key Results

  • Handling Complexity: When the action space was small (32 possible actions), standard DQN and BDQ performed similarly. However, when the action space exploded to ~2 million actions (at higher map resolutions), DQN failed to train, while BDQ continued to improve.
  • Training Efficiency: The Federated RL approach achieved significantly higher rewards (better data sharing) in the same amount of training time compared to non-federated approaches.
  • Performance vs. Oracle: The trained RL agents were compared against an "Oracle" (a theoretical perfect solver). The gap between the RL agent and the Oracle was remarkably small, proving the agents learned highly efficient policies.
  • Reliability: Trained agents consistently outperformed random (non-trained) agents, providing much higher "satisfaction" across all tested vehicle densities ($N=4, 6, 12$).

Limitations and Future Work

  • 2D vs. 3D: The current model uses a 2D Quadtree. While efficient, real-world LiDAR is 3D. The authors mention Octrees (the 3D version of Quadtrees) as a logical next step.
  • Single RSU: The paper focuses on a single junction. Future work could explore how vehicles hand off these cooperative sessions as they move between multiple RSUs.
  • Perfect Sensing Assumption: While the paper includes a sensor reliability variable ($\lambda_n$), it largely assumes that if a block is "occupied," that information is ground truth. Real-world sensor noise and "ghost objects" could add another layer of complexity to the RL reward function.

Summary updated

Abstract

Cooperative perception plays a vital role in extending a vehicle's sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared?, at which resolution?, and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperative perception messages (CPMs) by utilizing a quadtree-based point cloud compression mechanism. Furthermore, a federated RL approach is introduced in order to speed up the training process across vehicles. Simulation results show the ability of the RL agents to efficiently learn the vehicles' association, RB allocation, and message content selection while maximizing vehicles' satisfaction in terms of the received sensory information. The results also show that federated RL improves the training process, where better policies can be achieved within the same amount of time compared to the non-federated approach.
BibTeX
@article{abdelaziz2022vehicular,
  author = {Mohamed K. Abdel-Aziz and Cristina Perfecto and Sumudu Samarakoon and Mehdi Bennis and Walid Saad},
  title = {{Vehicular Cooperative Perception Through Action Branching and Federated Reinforcement Learning}},
  journal = {IEEE Trans. Communications},
  year = {2022},
  doi = {10.1109/tcomm.2021.3126650},
}

Loading related content...

Figure