BRNES: Enabling Security and Privacy-Aware Experience Sharing in Multiagent Robotic and Autonomous Systems

Md Tamjid Hossain, Hung Manh La, Shahriar Badsha, Anton Netchaev

Abstract

In decentralized Multiagent Reinforcement Learning (MARL), agents often utilize experience sharing (ES) to accelerate policy convergence. However, this collaborative mechanism introduces significant vulnerabilities, specifically Byzantine attacks where malicious agents provide misleading advice, and inference attacks where curious agents reconstruct an advisor's private strategies. This paper presents BRNES (Byzantine Robust Neighbor Experience Sharing), a decentralized framework designed to secure experience sharing through dynamic neighbor selection, weighted aggregation, and local differential privacy. Experimental results in predator-prey environments demonstrate that BRNES achieves up to an 8.32x speedup over existing methods in adversarial settings while effectively shielding agent privacy.

Introduction

Multiagent Reinforcement Learning (MARL) has become a cornerstone for coordinating autonomous robotic swarms. To overcome the slow convergence rates typical of individual learning, agents often employ "experience sharing" (ES), querying more experienced peers for guidance (e.g., Q-values). While effective in cooperative, trusted environments, ES creates two critical security gaps in decentralized systems. First, Byzantine attacks allow malicious actors to inject false data, leading agents into obstacles or away from objectives. Second, inference attacks allow curious agents to repeatedly query an advisor to reconstruct their private Q-tables, leaking sensitive strategic information.

Existing solutions often rely on centralized servers or assume a baseline of trust, both of which represent single points of failure or unrealistic assumptions in open robotic deployments. BRNES addresses these challenges by treating security and privacy as local, heuristic problems rather than global authentication tasks.

Methodology

The BRNES framework integrates three primary defensive pillars to ensure robust and private decentralized learning.

1. Adaptive Heuristic Neighbor Zones

Rather than maintaining static advisor-advisee relationships, BRNES utilizes dynamic "neighbor zones." An advisee calculates its zone based on environment dimensions and current agent density. Because agents are mobile and the zone is recalculated at each step, it becomes computationally and logistically difficult for a specific malicious attacker to "stalk" and consistently feed corrupted advice to a single target. This spatial randomness breaks the persistence required for many coordinated Byzantine strategies.

2. Weighted Experience Aggregation

To mitigate the impact of potentially malicious advice, BRNES implements a "skeptical" aggregation mechanism. Instead of blindly adopting external Q-values, the advisee uses a weight factor ($w$) to blend its own locally derived knowledge with the averaged advice from its current neighbors. This ensures that even if a portion of the neighborhood consists of Byzantine attackers, their influence is diluted by the agent's own experience and the presence of other honest neighbors.

3. Local Differential Privacy (LDP)

To protect against inference attacks, advisors apply Generalized Randomized Response (GRR) to their Q-values before transmission. This LDP mechanism adds mathematically calibrated noise to the shared data, providing "plausible deniability." An attacker receiving a value cannot determine with certainty whether it represents the advisor's true strategic belief or is a result of LDP-induced noise, thereby preventing the accurate reconstruction of the advisor's private Q-table.

Experimental Setup

The framework was evaluated using a modified predator-prey grid world domain across three distinct scales:

  • Small-scale: 5x5 grid with 5 agents and 1 obstacle.
  • Medium-scale: 10x10 grid with 10 agents and 3 obstacles.
  • Large-scale: 30x30 grid with 20 agents and 5 obstacles.

The researchers tested these environments under varying adversarial pressures, with the proportion of Byzantine attackers ranging from 0% to 40%.

Results and Discussion

Learning Performance in Adversarial Environments

BRNES demonstrated significant resilience in the face of active sabotage. In medium-scale environments with 40% attackers, BRNES was 8.32x faster at reaching the goal than the non-private AdhocTD framework. Furthermore, it outperformed the private DA-RL framework by a factor of 1.41x. While traditional LDP often introduces a "utility tradeoff" (where noise slows learning), BRNES shows that in adversarial worlds, this noise actually accelerates convergence by making the system robust against "optimal" false data injection.

Convergence Stability

BRNES achieved stable learning convergence in fewer than 200 episodes. In contrast, competing methods under the same adversarial pressure required between 400 and 900 episodes to reach similar stability, highlighting the efficiency of the weighted aggregation and dynamic zoning.

Privacy Protection

Without LDP protections, an inference attacker could reconstruct an advisor's strategy with over 70% accuracy within 3000 episodes. By implementing BRNES with a privacy budget of $\epsilon=0.1$, the success rate of these attacks dropped significantly, effectively shielding the advisor's private trajectories and preferences from curious peers.

Conclusion and Future Work

BRNES provides a robust decentralized framework for experience sharing in untrusted robotic swarms. By leveraging spatial randomness and local differential privacy, it secures the learning process against both malicious disruption and private data leakage.

Future research will focus on extending the framework to 3D continuous state spaces, which will require more complex distance and zone calculations. Additionally, the authors plan to investigate "selfish" agent behaviors—where agents are not purely malicious but seek to optimize their own rewards at the expense of the group—and adapt the framework for dynamic tasks where goals change over time.

Summary updated

BibTeX
@inproceedings{hossain2023brnes,
  author = {Md Tamjid Hossain and Hung Manh La and Shahriar Badsha and Anton Netchaev},
  title = {{BRNES: Enabling Security and Privacy-Aware Experience Sharing in Multiagent Robotic and Autonomous Systems}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341559},
}
Figure