Abstract
Distributed Deep Reinforcement Learning (DRL) at the edge offers a scalable approach to training autonomous systems through collaborative exploration. However, this decentralized architecture introduces a significant vulnerability: Byzantine agents. This paper investigates the impact of malfunctioning or malicious agents in discrete action spaces, demonstrating that even a small fraction of "bad actors"—as low as 6.25% of the total workforce—can completely derail the learning process. Our findings highlight the extreme sensitivity of discrete DRL environments to data poisoning and establish critical thresholds for system failure.
Introduction
As intelligence becomes ubiquitous, DRL models are increasingly deployed on edge devices such as robots and IoT sensors. These devices often collaborate to learn complex tasks, benefiting from distributed training that offers redundancy and speed. Yet, this reliance on multiple workers introduces the "Byzantine" problem: agents that behave unpredictably due to hardware failure, sensor noise, or adversarial interference. While previous research has explored network latencies and continuous control perturbations, the specific vulnerability of discrete action spaces—where agents must choose from a finite set of moves—remains under-analyzed. This study seeks to identify the breaking point of collaborative learning systems when faced with such malfunctions.
Methodology
The research utilizes the Synchronous Advantage Actor-Critic (A2C) algorithm within a hierarchical edge computing framework. In this architecture, multiple worker agents explore their environments in parallel, periodically uploading their experiences (actions and rewards) to an Edge Backend. The backend aggregates this data to calculate a global policy update, which is then synchronized back to all agents.
To simulate Byzantine behavior, the authors introduced agents that perform the inverse of the policy-recommended action. The experiments varied the number of malfunctioning agents and the frequency of their errors across a total pool of 16 workers. This allowed for an analysis of how the total volume of "poisoned" data affects the common policy.
Experimental Setup
The study benchmarked the system using two classic Atari 2600 environments via OpenAI Gym:
- PongNoFrameskip-v4: A task with 5 possible discrete actions.
- BreakoutNoFrameskip-v4: A task with 4 possible discrete actions.
These environments were selected for their clear reward signals and the precision required for successful task completion, making them ideal for testing the stability of DRL algorithms.
Key Results and Discussion
The Total Volume Rule
A primary insight from the study is that the distribution of errors is less critical than the total volume of Byzantine actions. The impact on the global policy remains consistent whether the errors originate from a single completely broken agent or are spread across multiple partially malfunctioning agents.
Sensitivity and Task Dependency
Discrete action spaces proved remarkably fragile. In the Pong environment, a 20% total Byzantine action rate prevented the system from learning entirely. While the system could converge at a 10% error rate, it remained highly unstable. In contrast, Breakout exhibited even higher sensitivity; the system failed to improve at a 10% error rate and required the error volume to drop to 2.5% for stable convergence.
The Threshold of Failure
The results suggest a hard limit for standard A2C aggregation. Once Byzantine actions cross a threshold (roughly 10-15% depending on task complexity), the gradient updates become sufficiently noisy to cancel out the progress of healthy agents. This "poisoning of the well" ensures the common policy never moves toward an optimal solution.
Conclusion
This research underscores the vulnerability of distributed DRL at the edge to Byzantine interference. The extreme sensitivity of discrete action spaces suggests that standard aggregation methods like simple averaging are insufficient for robust real-world deployment. Future work must focus on developing detection mechanisms to identify and isolate Byzantine agents and implementing robust statistical aggregation methods that can maintain policy integrity in the presence of outliers.
Summary updated