Using Cyber Terrain in Reinforcement Learning for Penetration Testing

Rohit Gangupantulu, Tyler Cody, Paul Park, Abdul Rahman, Logan Eisenbeiser, Dan Radke, Ryan ClarkCOINS 2021 · OpenReview

TL;DR

This paper introduces "cyber terrain" into reinforcement learning (RL) models for penetration testing. By incorporating elements like firewalls as obstacles, they make RL agents' attack strategies more realistic and aligned with how human adversaries operate, moving beyond purely vulnerability-score-based attack graphs.

What Problem Does This Paper Address?

Current approaches to using reinforcement learning (RL) for automated penetration testing often rely on attack graphs constructed solely from vulnerability scores (like CVSS). While this allows for scalability, it creates a disconnect from real-world attack scenarios. These models lack the "operational nuances" that human attackers consider, such as the impact of network infrastructure like firewalls, which act as obstacles. This leads to RL agents learning unrealistic attack campaigns that don't reflect how an adversary would actually plan and execute an attack. The paper aims to bridge this gap by incorporating "cyber terrain" concepts into RL environments for more realistic penetration testing.

Methods

The core contribution is a methodology for integrating "cyber terrain" into Markov Decision Process (MDP) models of attack graphs. They draw inspiration from the OAKOC (Observation and Fields of Fire, Avenues of Approach, Key Terrain, Obstacles and Movement, Cover and Concealment) framework used in military intelligence preparation of the battlefield.

Specifically, they propose two primary methods for incorporating cyber terrain, using firewalls as a concrete example:

  1. Adding via Reward: The reward function is modified to penalize agents for encountering firewalls. This is achieved by introducing a negative reward term that is protocol-dependent, making certain protocols (like FTP) more costly to traverse through a firewall than others (like SSH).
  2. Adding via State: The state transition probabilities are adjusted to reflect the presence and impact of firewalls. This involves modifying the state space and transition dynamics to account for firewalls as obstacles, influencing the agent's movement and decision-making.

They use a Deep Q-Network (DQN) with experience replay for training the RL agent. The attack graphs are initially constructed using the MulVal framework, which analyzes network representations.

Datasets

The paper does not explicitly mention using pre-existing public datasets for training or evaluation. Instead, they construct their own attack graph for a network with approximately 100 hosts, resulting in an attack graph with roughly 1000 vertices and 2300 edges. This network and its corresponding attack graph serve as their benchmark, which they note is significantly larger than those used in previous related work.

Key Insights and Ideas

  • Cyber Terrain is Crucial for Realistic RL Penetration Testing: Simply relying on vulnerability scores (CVSS) creates an incomplete picture. Incorporating network infrastructure like firewalls as "obstacles" significantly alters agent behavior and leads to more plausible attack paths.
  • Reward Engineering vs. State Modification: Both methods (modifying rewards or state dynamics) can introduce cyber terrain. Modifying rewards directly incentivizes avoiding obstacles, while modifying state dynamics changes the underlying environment's behavior, offering different ways to model terrain's impact.
  • Protocol-Specific Firewall Impact: The effectiveness of a firewall as an obstacle is not uniform; it depends on the protocols being blocked. This nuance is captured by assigning different penalty values or state transition impacts based on the protocol.
  • Larger-Scale Networks are Necessary: Previous RL penetration testing work used relatively small networks. This paper demonstrates their method on a significantly larger graph, highlighting the scalability of their approach.

Key Results

The paper presents quantitative results comparing three MDP configurations: a "vanilla" (terrain-blind) CVSS MDP, a reward-adjusted MDP, and a state-adjusted MDP.

  • Increased Hops: Both terrain-adjusted MDPs (reward and state) led to an increase in the total number of hops taken by the RL agent, indicating that agents had to navigate around firewalls.
  • Decreased Reward (Reward-Adjusted MDP): The reward-adjusted MDP showed a decrease in total reward despite taking more hops. This is attributed to the negative reward introduced by firewalls, confirming the reward engineering's effectiveness in penalizing firewall encounters.
  • Improved Progress (State-Adjusted MDP): The state-adjusted MDP showed a greater total reward, suggesting that by altering state dynamics, the agent could still make steady progress towards the terminal state while accounting for obstacles.
  • Qualitative Path Differences: Visualizations of the attack campaigns (Figure 4) clearly show that the terrain-adjusted agents take different paths, actively avoiding nodes identified as firewalls, whereas the vanilla agent does not. This visually confirms the impact of cyber terrain on attack strategy.

Limitations and Future Work

  • Limited Cyber Terrain Elements: The current work focuses primarily on firewalls as obstacles. The authors acknowledge that other aspects of cyber terrain (avenues of approach, key terrain, cover, concealment) are not yet incorporated.
  • Single Protocol Focus in Figures: While Table II shows results with all protocols available, Figure 3 plots average reward when the agent is restricted to a single protocol choice. This highlights the protocol-specific impact but limits direct comparison of overall performance across all protocols in the plots.
  • Future Work:
    • Expand Cyber Terrain Elements: Incorporate more OAKOC concepts into MDP construction.
  • Scale to Enterprise-Level Networks: Test the methodology on even larger, more complex enterprise networks to further validate scalability and realism.
  • Multiple Initial/Terminal States: Develop methods for using multiple starting and ending points to better map attack surfaces.
  • Transfer Learning and Meta-Learning: Explore how RL agents can generalize their penetration testing skills across different networks or learn more efficiently.

Summary updated

Abstract

Reinforcement learning (RL) has been applied to attack graphs for penetration testing, however, trained agents do not reflect reality because the attack graphs lack operational nuances typically captured within the intelligence preparation of the battlefield (IPB) that include notions of (cyber) terrain. In particular, current practice constructs attack graphs exclusively using the Common Vulnerability Scoring System (CVSS) and its components. We present methods for constructing attack graphs using notions from IPB on cyber terrain analysis of obstacles, avenues of approach, key terrain, observation and fields of fire, and cover and concealment. We demonstrate our methods on an example where firewalls are treated as obstacles and represented in (1) the reward space and (2) the state dynamics. We show that terrain analysis can be used to bring realism to attack graphs for RL.
BibTeX
@inproceedings{gangupantulu2021using,
  author = {Rohit Gangupantulu and Tyler Cody and Paul Park and Abdul Rahman and Logan Eisenbeiser and Dan Radke and Ryan Clark},
  title = {{Using Cyber Terrain in Reinforcement Learning for Penetration Testing}},
  booktitle = {COINS},
  year = {2021},
  url = {https://openreview.net/forum?id=9UCW4lqznE},
}
Figure