KGRGRL: A User's Permission Reasoning Method Based on Knowledge Graph Reward Guidance Reinforcement Learning

Lei Zhang, Pan Yu, Yi Liu, Qibin Zheng, Zhisong Pan

Abstract

Traditional cyberspace security assessments often fail to capture the full complexity of multi-domain environments, including physical and social factors. Existing permission reasoning methods typically rely on rigid, expert-defined rules that lack scalability and intelligence. This paper introduces KGRGRL, a novel method that combines Multi-Domain Cyberspace Knowledge Graphs (KG) with Reinforcement Learning (RL). By utilizing a reward guidance system based on 10 domain-specific rules, the proposed RL agent efficiently discovers permission paths across physical and digital domains. Experimental results demonstrate that KGRGRL achieves a 6% higher F1-score compared to the Translating Embedding (TransE) method, significantly improving the intelligence and accuracy of permission reasoning.

Introduction

In modern cybersecurity, determining user permissions is no longer a simple matter of checking access control lists. Real-world security is inherently multi-domain: physical access to a room can grant access to a terminal, which in turn provides network-level permissions. Most current reasoning methods are based on First-Order Logic or static rules, which are difficult to maintain as environments grow in complexity. The authors propose moving toward an automated, intelligent system that can "reason" through these complex relationships by navigating a unified Knowledge Graph.

Methodology: KG and Reward-Guided RL

The KGRGRL framework consists of two primary components. First, a Multi-Domain Cyberspace Knowledge Graph is constructed to provide a standard semantic description of entities such as physical spaces, devices, ports, services, and personnel. This graph allows the system to model dependencies across different domains (e.g., a person must be in a specific room to use a specific device).

Second, the system employs Reinforcement Learning (RL) to find permission paths. In this model, permissions are nodes in the graph, and the agent's goal is to identify all reachable permissions for a given user. To overcome the "sparse reward" problem common in RL, the authors implemented 10 reward setting rules. These rules guide the agent by providing intermediate rewards for logical steps, such as gaining control of a service after obtaining its password.

Optimal policy reasoning for permission nodes P1 through P5 within the reinforcement learning framework.

As shown in the figure above, the agent learns an optimal policy to navigate between permission states ($P_1$ to $P_5$), ensuring that the reasoning process is both efficient and logical.

Experimental Setup and Datasets

To evaluate KGRGRL, the researchers simulated a complex enterprise network environment. This simulation includes physical constraints (rooms) and network infrastructure (firewalls, routers, and servers).

The simulated multi-domain cyberspace environment used for training and evaluation, integrating physical rooms and network infrastructure.

The dataset involved 5,000 simulated users with varying initial permissions. The ground truth for these permissions was established using a traditional rule-based (BOR) method, allowing for a direct comparison of the RL agent's performance against established logic.

Key Results and Insights

The proposed KGRGRL method was compared against the Path Ranking Algorithm (PRA) and the Translating Embedding (TransE) method.

  • Accuracy: KGRGRL achieved an F1-score of 55.9%, outperforming TransE (49.97%) and PRA (37.24%).
  • Intelligence: Unlike rule-based systems that require manual updates for every new configuration, KGRGRL learns the underlying logic of the graph, making it more adaptable to changing environments.
  • Domain Integration: The success of the method underscores the importance of modeling the physical domain. By treating physical access as a prerequisite for digital access, the KG provides a more realistic representation of cybersecurity risks.

Conclusion

KGRGRL represents a significant step toward automated security reasoning. By combining the structured data of Knowledge Graphs with the exploratory power of Reinforcement Learning, the method provides a more comprehensive and intelligent way to assess user permissions. Future work will focus on scaling the system for real-world, large-scale enterprise environments and adapting to dynamic network changes in real-time.

Summary updated

Abstract

In general, multiple domain cyberspace security assessments can be implemented by reasoning user's permissions. However, while existing methods include some information from the physical and social domains, they do not provide a comprehensive representation of cyberspace. Existing reasoning methods are also based on expert-given rules, resulting in inefficiency and a low degree of intelligence. To address this challenge, we create a Knowledge Graph (KG) of multiple domain cyberspace in order to provide a standard semantic description of the multiple domain cyberspace. Following that, we proposed a user's permissions reasoning method based on reinforcement learning. All permissions in cyberspace are represented as nodes, and an agent is trained to find all permissions that user can have according to user's initial permissions and cyberspace KG. We set 10 reward setting rules based on the features of cyberspace KG in the reinforcement learning of reward information setting, so that the agent can better locate user's all permissions and avoid blindly finding user's permissions. The results of the experiments showed that the proposed method can successfully reason about user's permissions and increase the intelligence level of the user's permissions reasoning method. At the same time, the F1 value of the proposed method is 6% greater than that of the Translating Embedding (TransE) method.
BibTeX
@misc{zhang2022kgrgrl,
  author = {Lei Zhang and Pan Yu and Yi Liu and Qibin Zheng and Zhisong Pan},
  title = {{KGRGRL: A User's Permission Reasoning Method Based on Knowledge Graph Reward Guidance Reinforcement Learning}},
  year = {2022},
  eprint = {2205.07502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.07502},
}
Figure