Introduction
Automated Cyber Defence (ACD) represents a paradigm shift from reactive, manual security operations to proactive, adaptive systems capable of defending complex networks at machine speed. Reinforcement Learning (RL) and Multi-Agent Reinforcement Learning (MARL) are frequently cited as the most promising technologies for this transition. However, a significant gap exists between theoretical AI research and practical deployment. This paper serves as a strategic manual, providing guidelines for researchers and engineers to navigate the architectural decisions, algorithmic choices, and operational challenges inherent in applying RL to cybersecurity.
The Decision Framework: When to Use RL and MARL
A core contribution of this work is a structured decision-making framework designed to prevent the over-engineering of security solutions. The authors emphasize that machine learning is not a universal solution:
- Rules vs. RL: If a security problem has a clear, deterministic solution based on existing expert knowledge, rule-based systems remain the gold standard. RL should be reserved for scenarios requiring exploration and complex, multi-step planning in unknown or dynamic environments.
- Single-Agent vs. MARL: MARL introduces significant complexity and should not be used merely to speed up training. It is appropriate only for scenarios characterized by massive observation/action spaces, fragmented network architectures, or strict resilience requirements where distributed decision-making is a necessity.
Algorithmic Paradigms in Cyber Defence
The paper evaluates several RL/MARL paradigms through the lens of cyber constraints:
- Independent Learning (e.g., IPPO): Agents learn autonomously. While simple to implement, these methods often struggle with coordination in shared environments.
- Centralized Training with Decentralized Execution (CTDE): Methods such as MAPPO and QMIX allow agents to share global information during the training phase while maintaining independent action during deployment. This reduces communication overhead in real-world networks.
- Model-Based MARL: Algorithms like Dreamer or MBPO learn a "world model" to simulate interactions. This is particularly valuable in cybersecurity, where real-world training data is scarce, and the cost of failure during live training is prohibitively high.
Training Grounds: Simulation Environments
Because static datasets are often insufficient for training interactive agents, the paper highlights several open-source simulation environments that serve as the primary benchmarks for ACD:
- YAWNING TITAN: An abstract, graph-based simulation for high-level strategy.
- CyBORG (CAGE Challenges): A more detailed environment for single and multi-agent cyber scenarios.
- Specialized Simulators: Tools like IPMSRL and GVA focus on maritime and vehicle architectures, though these are often restricted in availability.
Practical Challenges and Pathologies
Deploying RL in cybersecurity is complicated by several "pathologies" that can lead to system failure:
- Relative Overgeneralization: Agents may converge on safe but sub-optimal behaviors because exploring more effective strategies independently often results in immediate penalties.
- The Moving Target Problem: In a network where both defenders and adversaries are learning, the environment becomes non-stationary. This makes it difficult for agents to converge on stable, reliable policies.
- The Sim-to-Real Gap: Real-world networks are messy. Policies trained in high-fidelity simulations often fail when faced with noisy logs, false-positive alerts, or unavailable actuation points in a live production environment.
- Reward Shaping: Designing reward functions is a "minefield." Sparse feedback (e.g., only knowing a breach occurred after the fact) can lead agents to "game" the system, finding ways to maximize rewards without actually improving security posture.
Future Directions
The authors identify several critical areas for future research to make MARL viable for enterprise-grade security:
- General-Sum Games: Moving beyond purely cooperative defense to model scenarios where different network stakeholders may have competing interests.
- Robust Generalization: Developing agents that can maintain performance against novel, unseen adversary behaviors.
- Human-Machine Teaming: Shifting from full autonomy to "recommender" systems that keep human experts in the loop, necessitating improvements in AI explainability and trust.