Abstract
Next-generation communication networks rely on "Cooperative Intelligence" (CI) where diverse agents, such as drones and edge controllers, collaborate to solve complex tasks. While Multi-Agent Reinforcement Learning (MARL) is a primary driver for this cooperation, it often requires sharing sensitive local data, leading to significant privacy risks. This paper introduces PP-MARL, a novel framework that integrates hierarchical split learning, homomorphic encryption, and differential privacy. PP-MARL protects sensitive agent data while reducing communication bandwidth by up to 91% compared to state-of-the-art methods, ensuring efficient and secure cooperation in 5G/6G environments.
The Privacy Challenge in Cooperative Intelligence
In modern network environments, devices must work together for tasks like traffic routing and resource allocation. However, standard MARL approaches require agents to share local observations, actions, and rewards to learn effectively. In real-world scenarios, this data—such as a drone's trajectory or a user's precise location—is highly sensitive.
Existing solutions face significant hurdles. Federated Learning (FL) often assumes homogeneous agents, which is unrealistic for diverse network hardware. Standard encryption methods are typically too slow and bandwidth-intensive for high-speed 5G/6G requirements. Furthermore, simple data masking remains vulnerable to inference attacks, where adversaries reverse-engineer private data from shared gradients.
The PP-MARL Framework
To address these challenges, PP-MARL employs a three-pronged approach to balance privacy, speed, and intelligence within the communication environment.
Hierarchical Critics and Split Learning
Instead of transmitting raw, high-dimensional data to a central server, each agent utilizes a "local critic." This model compresses sensitive observations into low-dimensional "q-values." This split learning approach ensures that the most sensitive raw data never leaves the local device.
Interactive Homomorphic Encryption (HE)
To protect the transmitted q-values, PP-MARL uses HE. Because the q-values are already compressed, the computational overhead of encryption is significantly lower than encrypting raw sensor or video data, making it feasible for real-time applications.
Differential Privacy (DP)
Calibrated Gaussian noise is added to the loss function during training. This mathematical safeguard ensures that even if an attacker intercepts training updates, they cannot reliably infer whether specific private information was part of the training set.
Performance and Key Results
The researchers evaluated PP-MARL using two scenarios: a drone-assisted communication simulation and a network control environment (HSD-IoV) using real-world vehicle mobility traces from the SNDlib dataset.
- Bandwidth Efficiency: PP-MARL achieved an 84% to 91% reduction in bandwidth compared to state-of-the-art cooperative schemes like FedQ and FACMAC.
- Superior Privacy: The framework provided 1.1 to 6 times better privacy protection against inference attacks than competing models.
- Reduced Latency: In Internet of Vehicles (IoV) tests, PP-MARL reduced mean control delays by up to 72.6% compared to non-intelligent or centralized baselines.
- Mission Success: In drone coverage tasks, PP-MARL achieved higher performance gains (9.4) than standard MADDPG (6.9), proving that privacy does not have to come at the cost of utility.
Insights and Future Outlook
A key insight of this work is the "HE-friendly" architecture. By moving heavy data processing to local agents and only encrypting low-dimensional outputs, the authors bypassed the latency issues that typically hinder Homomorphic Encryption in reinforcement learning. Furthermore, the results demonstrate that "private cooperation" is more effective than "selfish intelligence," as PP-MARL significantly outperformed non-cooperative methods.
While highly efficient, the framework currently requires agents to have sufficient power for encryption/decryption tasks, making it better suited for edge devices than low-power sensors. Future work will focus on optimizing interactive encryption to further reduce communication rounds and exploring polynomial activation functions to improve learning performance in increasingly complex environments.
Summary updated