Abstract
Reinforcement learning (RL) operating on attack graphs leveraging cyber terrain principles are used to develop reward and state associated with determination of surveillance detection routes (SDR). This work extends previous efforts on developing RL methods for path analysis within enterprise networks. This work focuses on building SDR where the routes focus on exploring the network services while trying to evade risk. RL is utilized to support the development of these routes by building a reward mechanism that would help in realization of these paths. The RL algorithm is modified to have a novel warm-up phase which decides in the initial exploration which areas of the network are safe to explore based on the rewards and penalty scale factor.
Introduction
In the lifecycle of a cyber-attack, reconnaissance is often the most critical yet hardest-to-detect phase. Adversaries must navigate a network to identify "crown jewel" assets and exploitable services while remaining under the radar of security operations centers (SOCs). This paper introduces SDR-RL, a reinforcement learning framework designed to automatically discover "Surveillance Detection Routes" (SDRs). These are the stealthy paths an attacker might take to map a network while minimizing the risk of triggering defensive alarms.
By modeling a network as "cyber terrain"—where different services and hosts carry varying levels of defensive risk—the researchers provide a method for defenders to preemptively identify and secure the most likely paths of intrusion.
Methodology
Double Agent Architecture (DAA) and DA-PPO
The SDR-RL framework builds upon the Double Agent Architecture (DAA). In this model, the task of navigating the network is split between two specialized components: a structural agent that learns the underlying network topology and an exploiting agent that selects specific actions based on that topology.
To improve upon previous iterations, the researchers implemented DA-PPO, replacing the standard Advantage Actor-Critic (A2C) algorithm with Proximal Policy Optimization (PPO). PPO provides significantly greater training stability and faster convergence, which is essential when dealing with the high-dimensional state spaces found in complex enterprise networks.
The Novel Warm-up Phase
A primary innovation of this work is the introduction of a Warm-up Phase in the RL training cycle. During this phase, the agent explores the network environment without updating its neural network weights. The goal is to identify "safe" nodes—areas of the network that provide positive rewards with minimal risk. These nodes are then dynamically added to the agent's goal set for the subsequent training phase. This approach mimics the intuition of a human penetration tester who "feels out" a network's perimeter before committing to a specific lateral movement strategy.
Cyber Terrain Modeling
The framework utilizes "cyber terrain" principles to define the reward function. Not all network nodes are equal; services like authentication servers or internal databases carry higher "penalties" (risk of detection) than common HTTP services. By adjusting a penalty scale factor, the researchers can simulate different adversary profiles:
- Low Penalty Scale: Simulates a "noisy" or reckless attacker who scans broadly and quickly.
- High Penalty Scale: Simulates a risk-averse, nation-state actor who prioritizes stealth and takes the path of least resistance.
Experimental Setup
Due to the sensitive nature of network vulnerability data, the study utilized a custom-simulated cyber range. This environment was designed to reflect real-world enterprise complexity, featuring:
- Layered defensive zones, including DMZs and private internal subnets.
- Firewalls with strict access control lists (ACLs).
- A variety of host configurations running services such as email, web servers, and database protocols.
Key Results
Performance and Efficiency
The transition to DA-PPO yielded substantial performance gains. The DA-PPO model converged on optimal surveillance routes in less than 10 minutes, whereas the older DA-A2C model required over 30 minutes and significantly more training episodes to reach the same level of proficiency.
Behavioral Adaptation
The agent demonstrated a clear ability to adapt its behavior based on the perceived risk of the cyber terrain. As the penalty for interacting with high-value defensive nodes increased, the RL agent naturally shifted its strategy. In high-penalty scenarios, the agent identified routes that touched the minimum number of services necessary to reach the objective, effectively finding the "stealthiest" path through the network.
Scalability
The DAA modeling approach proved resilient to increases in network complexity. Even as more nodes and more stringent penalties were added to the environment, the agent maintained its ability to identify efficient surveillance routes, suggesting the framework is viable for larger enterprise-scale simulations.
Conclusion and Future Work
SDR-RL represents a significant step forward in automated red-teaming and defensive path analysis. By combining RL with cyber terrain modeling and a novel warm-up exploration phase, the framework provides a realistic simulation of adversary reconnaissance behavior.
Future research will focus on moving beyond simulated environments to test the model against live network traffic. Additionally, the authors plan to integrate more granular vulnerability data, such as the Common Vulnerability Scoring System (CVSS3) and Common Weakness Enumeration (CWE), to further refine the reward and penalty mechanisms, making the simulations even more representative of real-world cyber warfare.
Summary updated