Adversarial Deep Reinforcement Learning Based Adaptive Moving Target Defense

Taha Eghtesad, Yevgeniy Vorobeychik, Aron Laszka

Introduction

Moving Target Defense (MTD) is a proactive security paradigm that continuously alters a system's configuration to increase uncertainty and complexity for attackers. While effective, MTD faces a significant optimization challenge: changing configurations too slowly allows attackers to succeed, while changing them too quickly disrupts system availability and wastes resources. This paper introduces a Deep Reinforcement Learning (DRL) framework to automate these decisions, modeling the interaction as a partially observable game where agents learn to balance security and performance.

The MTD Problem: A Balancing Act

Traditional cybersecurity is often reactive, focusing on patching vulnerabilities after they are discovered. MTD flips this script by rotating IP addresses, shifting cloud instances, or altering software configurations to force attackers into perpetual reconnaissance. However, finding the "Goldilocks" zone of configuration frequency is difficult for human administrators. This research addresses the need for an automated, adaptive engine capable of navigating complex, uncertain environments to find optimal defense policies.

Methodology

The authors frame the MTD challenge as a Partially Observable Markov Decision Process (POMDP). In this scenario, the defender has imperfect information regarding server compromises, and the adversary lacks full visibility into the defender's reimaging schedule.

The Reinforcement Learning loop between the Agent and the Environment.

Technical Innovations

  • Deep Q-Network (DQN): The framework utilizes a neural network to estimate the value of actions, such as reimaging or probing a server.
  • Compact History Representation: To address partial observability without the computational cost of Recurrent Neural Networks (like LSTMs), the authors introduced two time-based features: time since the last probe and time since the last reimage. This "compact memory" allows the agent to make informed decisions based on temporal patterns.
  • DQN Enhancements: The implementation includes Double-DQN to prevent value overestimation, Dueling-DQN to separate state value from action advantage, and Prioritized Experience Replay for more efficient learning.

Experimental Setup

The researchers implemented the game model as a custom environment in OpenAI Gym, allowing agents to generate data through millions of simulated interactions. The simulation parameters were derived from established game-theoretic literature to ensure realistic modeling of attack costs and server dynamics.

Baseline values and symbols for the environment, heuristics, and RL parameters.

Key Insights and Results

  • The Power of Memory: The study demonstrated that simple time-based counters are highly effective for decision-making in hidden games. This compact memory representation outperformed more complex architectures while remaining computationally efficient.
  • Superiority Over Heuristics: The DRL-based defender consistently achieved higher average rewards than traditional heuristic strategies like "MaxProbe" or "Control-Threshold." Unlike static policies, the DRL agent learned to be unpredictable, thwarting timed attacks.
  • Emergent Deception: Interestingly, the DRL-trained adversary developed deceptive tactics, such as intentionally avoiding certain servers to lull the defender into a false sense of security before striking.
  • Robustness and Viability: The agent maintained high security even when sensors were noisy (detecting probes only 50% of the time). Furthermore, the models were trained in just a few hours on standard consumer-grade hardware, proving the approach's practical feasibility.

Limitations and Future Work

While promising, the current model primarily trains against fixed heuristic opponents. Future research could employ Policy-Space Response Oracles (PSRO) to train agents against evolving, learning adversaries. Additionally, scaling the framework to multi-agent distributed systems and transitioning from simulations to high-fidelity network emulators remain key areas for development.

Summary updated

Abstract

Moving target defense (MTD) is a proactive defense approach that aims to thwart attacks by continuously changing the attack surface of a system (e.g., changing host or network configurations), thereby increasing the adversary’s uncertainty and attack cost. To maximize the impact of MTD, a defender must strategically choose when and what changes to make, taking into account both the characteristics of its system as well as the adversary’s observed activities. Finding an optimal strategy for MTD presents a significant challenge, especially when facing a resourceful and determined adversary who may respond to the defender’s actions. In this paper, we propose a multi-agent partially-observable Markov Decision Process model of MTD and formulate a two-player general-sum game between the adversary and the defender. To solve this game, we propose a multi-agent reinforcement learning framework based on the double oracle algorithm. Finally, we provide experimental results to demonstrate the effectiveness of our framework in finding optimal policies.
BibTeX
@inproceedings{eghtesad2020adversarial,
  author = {Taha Eghtesad and Yevgeniy Vorobeychik and Aron Laszka},
  title = {{Adversarial Deep Reinforcement Learning Based Adaptive Moving Target Defense}},
  booktitle = {GameSec},
  year = {2020},
  doi = {10.1007/978-3-030-64793-3\_4},
}
Figure