Evolving Reinforcement Learning Environment to Minimize Learner's Achievable Reward: An Application on Hardening Active Directory Systems

Diksha Goel, Aneta Neumann, Frank Neumann, Hung Nguyen, Mingyu Guo

Introduction

Active Directory (AD) serves as the identity management backbone for approximately 90% of Fortune 1000 companies. Its complexity, however, makes it a primary target for "identity snowball" attacks, where an adversary traverses a graph of accounts and computers to escalate privileges to a "Domain Admin" (DA) node. Defending these massive networks is a significant challenge due to scalability issues and the need to anticipate strategic, adaptive attackers.

This paper introduces a novel framework for hardening AD systems by modeling the security struggle as a Stackelberg game. By utilizing Reinforcement Learning (RL) to simulate a strategic attacker and a "Critic network assisted Evolutionary Diversity Optimization" (C-EDO) to evolve defensive configurations, the authors achieve superior performance and scalability compared to previous state-of-the-art methods.

The C-EDO Framework

The researchers frame the hardening problem as a game where the defender (the leader) configures the environment by revoking access rights (edges), and the attacker (the follower) attempts to find the most successful path.

  • The Attacker (RL): The attacker employs Proximal Policy Optimization (PPO). While the "Actor" decides on exploration paths, the "Critic" network estimates the success probability of the current state.
  • The Defender (C-EDO): Instead of seeking a single optimal defense, the defender maintains a diverse population of configurations. The core innovation lies in using the RL agent's own Critic network as a fitness function. This allows the defender to identify which network configurations are objectively most difficult for the attacker to navigate.
The C-EDO framework illustrating the interaction between the RL attacker and the evolutionary defender.

By training the attacker against multiple environments in parallel, the RL agent gains shared experience, leading to faster convergence and a more robust, generalized policy.

Diversity as a Strategic Defense

A central insight of this work is that diversity prevents the RL attacker from overfitting to a single network configuration. By evolving a diverse population of environments, the defender forces the attacker to learn a strategy that is effective across a range of scenarios, which in turn provides a more accurate measure of the network's true resilience.

The process of maximizing blocked edge diversity within the C-EDO algorithm.

This symbiotic relationship between the attacker's learning and the defender's evolution ensures that as the attacker becomes more sophisticated, the Critic provides an increasingly accurate "score" for the defender's configurations.

Experimental Setup and Datasets

To test the framework, the authors used DBCREATOR to generate synthetic AD graphs of varying sizes:

  • r500 to r4000: Ranging from 1,500 nodes (3,500 edges) to 12,000 nodes (45,800 edges).
  • Edge Types: The simulations included critical AD relationships such as HasSession, MemberOf, and AdminTo.
  • Probabilistic Modeling: Different edge properties (failure, detection, and success rates) were simulated to reflect real-world uncertainties.

Key Results and Scalability

The C-EDO approach demonstrated significant improvements over existing baselines:

  • Scalability: The method successfully scaled to the r4000 graph (12,000 nodes), whereas the previous state-of-the-art (NNDP-EDO) failed beyond the r2000 level.
  • Defensive Efficacy: In the r1000 graph, C-EDO reduced the attacker's average success rate to 41.01%, outperforming the previous best of 42.77%.
  • Attacker Realism: The RL-based attacker proved to be a more formidable model than previous value-iteration-based methods, achieving higher success rates and thus providing a more rigorous test for defensive strategies.

Limitations and Future Directions

While the framework marks a significant advancement, several areas for future work remain. The current model assumes a fixed budget for blocking edges and uses static graphs. Real-world AD environments are dynamic, with users logging in and out constantly. Future research will aim to integrate dynamic graph modeling and more complex Intrusion Detection System (IDS) simulations to further enhance the realism of the security game.

Summary updated

BibTeX
@inproceedings{goel2023evolving,
  author = {Diksha Goel and Aneta Neumann and Frank Neumann and Hung Nguyen and Mingyu Guo},
  title = {{Evolving Reinforcement Learning Environment to Minimize Learner's Achievable Reward: An Application on Hardening Active Directory Systems}},
  booktitle = {GECCO},
  year = {2023},
  url = {https://openreview.net/forum?id=Rzioj8TO_7},
}
Figure