TL;DR
FOE-Dreamer is a model-based reinforcement learning framework that trains autonomous cyber defense agents directly in operational networks, bypassing the "fidelity gap" of traditional simulators. By factoring the world model into defender-driven, user-driven, and attacker-driven components, it achieves 2x better performance than standard baselines within a strict 3-day training budget.
What Problem Does This Paper Address?
Most autonomous cyber defense research relies on simulators because deep reinforcement learning (RL) typically requires millions of trial-and-error interactions. However, simulators often fail to capture the messy realities of real networks—such as authentication delays, partial action failures, and the unpredictable rhythm of benign user traffic. This creates a "fidelity gap," where policies that work perfectly in simulation often degrade or fail when transferred to real hardware.
Training directly in a real network would solve this, but it is prohibitively slow and expensive. Every exploratory action (like rebooting a server or isolating a host) carries an operational cost. The authors tackle this by applying model-based RL, which allows an agent to "dream" or plan inside a learned internal model of the environment, drastically reducing the number of real-world interactions needed to learn effective defense strategies.
Methods
The core contribution is FOE-Dreamer (Factored Opponent-Enhanced Dreamer). It builds on the state-of-the-art Dreamer architecture but introduces two critical innovations for the cybersecurity domain:
- Factored World Model: Instead of one giant "black box" state, FOE-Dreamer splits its internal world model into three distinct latent components:
- Controllable Latent ($z_t$): Captures host states directly influenced by defender actions (e.g., isolation status, deception deployment).
- Exogenous Latent ($u_t$): Captures background noise from benign users (e.g., service load, periodic maintenance).
-
Opponent Embedding ($\tau$): Inferred from observations to summarize the attacker's behavior and intent.
This separation prevents the agent from getting confused by background noise, leading to much more stable policy gradients during training.
-
Opponent Modeling: The agent uses a recurrent encoder to infer the "type" of attacker it is facing (e.g., a stealthy "Avoidant" attacker vs. an aggressive "Enticed" one) based on how they react to defensive moves like honeypots.
Datasets and Testbed
Rather than using abstract gym environments, the authors developed a high-fidelity 8-host OpenStack testbed that emulates a mid-size enterprise network.
- Real Software & Substrate: The environment runs real Ubuntu VMs on a cloud substrate, featuring services like WordPress, SSH, and PostgreSQL.
- Vulnerabilities: The testbed includes four classes of real-world vulnerabilities, including high-severity remote code execution (CVE-2023-6553).
- User Emulation: The GHOSTS framework generates human-correlated traffic (browsing, admin sessions, file transfers) that overlaps with attacker activity.
- Interface: A dedicated C2 server bridges the RL agent's abstract tactical decisions to concrete network operations via gRPC and OpenStack APIs.
Key Insights and Ideas
- Factoring as a Variance Reducer: By separating user noise from defender actions, the authors found that the policy gradient variance drops significantly. The agent no longer "blames" its own actions for alert spikes caused by benign user activity.
- Opponent Inference for Deception: Knowing if an attacker is "Avoidant" allows the agent to use cheap deception (like honeypots) effectively to trigger a withdrawal. If the attacker is "Enticed," the agent learns it must move straight to decisive containment.
- Sample Efficiency is the Binding Constraint: FOE-Dreamer demonstrates that high-fidelity training is possible within a 3-day wall-clock budget on a single GPU, a feat that model-free methods like Rainbow or IQN could not match under the same constraints.
Key Results
- 2x Performance Gain: FOE-Dreamer achieved roughly 2x better episode loss compared to Rainbow and IQN baselines.
- Surpassing Human-Inspired Heuristics: Within three days of live training, the learned agent outperformed an "Expert-Inspired Heuristic" based on standard cyber-deception playbooks.
- Latent Validation: Linear decoding tests confirmed the factoring worked as intended: the exogenous latent ($u_t$) accurately tracked user activity ($R^2$ up to 0.79), while the controllable latent ($z_t$) focused on defender actions (0.71 accuracy).
Limitations and Future Work
- Scaling: The current testbed is limited to 8 hosts. Scaling this approach to enterprise-level networks with 50+ hosts is a significant deployment challenge.
- Adaptive Adversaries: The evaluation used scripted attacker profiles. Future work is needed to test against learning adversaries that can adapt to the defender's strategy.
- The "Final" Gap: While OpenStack is much closer to reality than a simulator, it is still an emulation. Transferring these policies to a production network with real human users and zero-day threats remains the ultimate goal.
Summary updated