Multiple Domain Cyberspace Attack and Defense Game Based on Reward Randomization Reinforcement Learning

Lei Zhang, Yu Pan, Yi Liu, Qibin Zheng, Zhisong Pan

Abstract

The existing network attack and defense method can be regarded as game, but most of the game only involves network domain, not multiple domain cyberspace. To address this challenge, this paper proposed a multiple domain cyberspace attack and defense game model based on reinforcement learning. We define the multiple domain cyberspace include physical domain, network domain and digital domain. By establishing two agents, representing the attacker and the defender respectively, defender will select the multiple domain actions in the multiple domain cyberspace to obtain defender's optimal reward by reinforcement learning. In order to improve the defense ability of defender, a game model based on reward randomization reinforcement learning is proposed. When the defender takes the multiple domain defense action, the reward is randomly given and subject to linear distribution, so as to find the better defense policy and improve defense success rate. The experimental results show that the game model can effectively simulate the attack and defense state of multiple domain cyberspace, and the proposed method has a higher defense success rate than DDPG and DQN.

Introduction: The Multi-Domain Challenge

Traditional cybersecurity research often treats the "battlefield" as a purely logical network domain, focusing on packets, protocols, and software vulnerabilities. However, modern sophisticated attackers do not limit themselves to the digital realm; they may physically enter a facility, compromise specific hardware terminals, or exploit stolen digital credentials.

Existing game-theoretic models are frequently too narrow to capture these multi-layered realities. Furthermore, when training automated defenders using Reinforcement Learning (RL), agents often fall into the trap of "suboptimal policies"—finding a strategy that is "good enough" to stop basic attacks but failing to discover the most robust defense against diverse, multi-stage threats. This paper proposes a holistic game model that spans the physical, network, and digital domains to bridge the gap between simplistic simulations and modern security challenges.

Proposed Game Model

The researchers constructed a zero-sum game between two competing agents: an Attacker and a Defender. The environment is modeled across three distinct layers:

  1. Physical Domain: Includes physical locations (rooms, buildings) and hardware terminals.
  2. Network Domain: Encompasses interfaces, traffic paths, and firewall configurations.
  3. Digital Domain: Covers information assets like usernames, passwords, and encrypted files.
Overview of the multi-domain cyberspace game model, illustrating the interaction between attacker and defender agents across physical, network, and digital domains.

As illustrated in the model, the defender observes the state of the cyberspace and selects actions from a multi-domain action list (e.g., restarting a server, controlling a terminal, or deleting firewall ACLs) to maximize their reward. The attacker simultaneously attempts to navigate the topology to reach sensitive targets.

Reward Randomization Reinforcement Learning (RR-DDPG)

To navigate the complex, continuous action spaces of a multi-domain environment, the authors utilized the Deep Deterministic Policy Gradient (DDPG) algorithm, which operates on an Actor-Critic framework.

The core innovation of this paper is the introduction of Reward Randomization. In standard RL, agents can become "lazy" once they find a moderately successful strategy. By occasionally sampling the defender's reward from a linear distribution rather than providing a fixed value, the authors introduce "noise" that forces the agent to explore the environment more thoroughly. This stochastic interference prevents the agent from settling on a mediocre strategy too early, helping it discover truly optimal and resilient defense policies.

Experimental Evaluation

The model was evaluated using a custom-built simulation environment featuring a topology of five distinct areas, including computers, firewalls, routers, switches, and servers. The dataset consisted of 100 generated episodes, where 100 attackers were randomly generated in each episode to attempt multi-step breaches (e.g., entering a room to gain a firewall password to eventually access a sensitive file on a server).

The researchers compared their Reward Randomization DDPG (RR-DDPG) against standard DDPG and Deep Q-Networks (DQN). The primary performance metric was the Assault Success Rate (ASR).

Comparison of Assault Success Rate (ASR) across 100 episodes for DQN, DDPG, and the proposed RR-DDPG model.

As shown in the results, the RR-DDPG model achieved the lowest ASR, successfully blocking more attackers than the baseline models. While standard DDPG improved over time, the randomized version found superior policies that the standard version missed. Additionally, the Defense Reward (DR) steadily increased throughout the training, confirming that the defender learned to effectively mitigate threats in a multi-domain context.

Conclusion and Future Work

By integrating physical and digital domains into a unified game model, this research provides a more realistic framework for automated cyber-defense. The use of reward randomization proved effective in enhancing the exploration capabilities of RL agents, leading to higher defense success rates.

Future work will look to expand the model by including the Social Domain (e.g., social engineering) and testing the scalability of these agents in massive, enterprise-scale environments with thousands of nodes. Furthermore, the assumption of "bounded rationality" will be challenged by modeling more unpredictable and irrational attacker behaviors.

Summary updated

Abstract

The existing network attack and defense method can be regarded as game, but most of the game only involves network domain, not multiple domain cyberspace. To address this challenge, this paper proposed a multiple domain cyberspace attack and defense game model based on reinforcement learning. We define the multiple domain cyberspace include physical domain, network domain and digital domain. By establishing two agents, representing the attacker and the defender respectively, defender will select the multiple domain actions in the multiple domain cyberspace to obtain defender's optimal reward by reinforcement learning. In order to improve the defense ability of defender, a game model based on reward randomization reinforcement learning is proposed. When the defender takes the multiple domain defense action, the reward is randomly given and subject to linear distribution, so as to find the better defense policy and improve defense success rate. The experimental results show that the game model can effectively simulate the attack and defense state of multiple domain cyberspace, and the proposed method has a higher defense success rate than DDPG and DQN.
BibTeX
@misc{zhang2022multiple,
  author = {Lei Zhang and Yu Pan and Yi Liu and Qibin Zheng and Zhisong Pan},
  title = {{Multiple Domain Cyberspace Attack and Defense Game Based on Reward Randomization Reinforcement Learning}},
  year = {2022},
  eprint = {2205.10990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.10990},
}
Figure